Building automation framework is important for starting the automation project, and it’s one of the steps which impacts the quality of the automation script. A good automation framework will help us to save time on implementing as well as maintaining the test script. In this article, we’ll share about the minimum requirements for an automation test framework based on our experience with a lot of automation projects.
1. Automation Framework
Firstly, we need to understand what an automation framework is. The automation framework includes the tools, libraries, the way we organize source code and the guidelines to implement or execute test cases. The benefits of building a good automation framework includes:
- Usability: Not only the technical guys but also the non-technical can get acquainted and implement the test case easily with a good automation framework. Some automation tools or frameworks support the code-gen feature so that the user can create the test case quickly. Besides, they can easily configure the browser, test environment or the test suite they want to run.
- Maintainability: By organizing the automation project and applying OOP, Design Pattern into the automation framework will help us to avoid duplicated code and save a lot of effort on updating the test script when there’s something changed.
- Test coverage: Automation test cases can be executed quickly, and they don’t require human effort while executing. An automation framework can support us to run tests on crossed browsers or platforms, so it can help us cover a lot of test environments. Besides, we can also apply data-driven in our automation test framework to cover more test cases.
- Extensibility: A good automation testing framework allows us to add new browsers or test environments easily.
- Report: After running test cases, a good report will be generated automatically. It can show the detailed information of the test run accurately.
Here are some kinds of popular automation frameworks:
- Linear
- Modular
- Data-driven
- Keyword-driven
- Behavior-driven Development
- Hybrid
Until now, Behavior-driven Development and Hyrid are used most in our projects.
2. Basic requirements for an automation framework
Based on our experience in building automation frameworks, here is the minimum list of requirements for an automation framework.
2.1. Well organization
The new member can get acquainted with a well-organized automation framework quickly. By storing source code in specific folders, we also can maintain or add new code easily. It also helps us to check if the code we want to add has been implemented and avoid duplicated code.
Based on the above picture, we can know where we store the core of the framework, test data, page object, test case of this project.
2.2. Support multiple platforms, crossed browsers and test environments
Based on our experience, we usually need to run tests on various kinds of platforms, browsers and test environments, so we need the mechanism to change them easily. We can define them in the configuration file, command line argument or through the system variables so that users don’t need to modify code when they want to change these configurations.
2.3. Apply Page Object Model design pattern
Page Object Model is one of the most popular design patterns in web automation. All the methods and locators related to a web page will be stored in a class. It helps us to avoid duplicated code and makes our framework easier to maintain.
2.4. Apply data-driven
Currently, most test runners like Test NG, NUnit, Pytest, Cucumber or Specflow also support us to implement data-driven in our test framework. We can leverage this feature to apply for the test cases which have the same steps, but the input data is different.
2.5. Run tests in parallel
For the project which has a huge number of test cases, we need our automation framework to be able to run tests in parallel so that we can reduce test execution time and have the product quality status as early as possible. It depends on the test runner you choose. Let’s see the below picture for the example of TestNG suite.
2.5. Filter test case for running tests
In the real project, we will define many test suites like regression test, smoke test or a specific suite for a feature. Therefore, our automation framework should support us in organizing the test suites and running the necessary test case only to reduce time for test execution. With various kinds of test runners, they can help us filter test cases by tag, annotation, attribute or title. Here is the picture for filter test cases with Pytest.
2.6. Integrate with API/Database
Our framework should be ready to work with API or Database so that we can do the API or Database testing. Moreover, if we only focus on UI test cases, we can leverage the API/Database to prepare or clean test data. Thanks to this, our test script will be more stable, and time for test data manipulation will be shorter.
2.7. Logging
Logging is very important for debugging. One of the biggest challenges of automation test is the flaky test. For the flaky test, we need to spend a lot of effort for analyzing it, especially running tests on the CI/CD tools like Jenkins or Azure DevOps pipeline. Efficiency logging will help us investigate the issue quickly.
2.8. Reporting
After running tests, our test automation framework should generate a test report automatically. The report should show detailed information of the test runs like the number of passed/failed/skipped test cases. For the failed test case, we should add the screenshot to the report. A good report can show us the history and statistics of the test cases. Currently, there are a lot of report tools we can leverage like Extent, Allure, Report Portal.
2.9. Integrate with CI/CD tools
For having the product quality status and finding bugs as early as possible, we need to apply continuous testing in our project. Therefore, integrating with the CI/CD tool is a must-have requirement now. We can trigger the test automatically at the scheduled time or after a new build release.
2.10. Guideline
After finishing building an automation framework, we should create a guideline so that the new members can install, get acquainted and use it to implement the new test cases quickly.
Conclusion
After finishing all the above features, I believe that we can start a new automation project. Of course, based on a specific project, we need to customize or add more features like visual validation, integration with test management and bug tracking systems like Jira, Zepher, TestRail or alert mechanism, etc. For now, AI is becoming popular, so we should consider AI plugins/libraries and add them into your automation framework. The automation framework is important, so we should treat it as another product besides the main product of our project and spend effort on it.