Key takeaways:
The testing phase, in a CI/CD pipeline holds importance as it ensures software quality, mitigates risks offers feedback to developers and ultimately delivers dependable and valuable software to end users. It plays a role, in the triumph of software development projects and the overall contentment of stakeholders.
Applying best practices for the testing stage in CI/CD pipelines is a crucial component of modern software development. It enhances software quality, reduces risks, improves efficiency, and ultimately contributes to the success of software projects and the satisfaction of end-users.
This article explores the recommended best practices for the testing stage in CI/CD pipelines.
Automate all of your testing
The primary objective of unit, integration, and functional tests is to enhance confidence in every new release deployed. In principle, an extensive suite of tests ensures that new features are published without introducing regressions.
The main aim of unit, integration, and functional tests is to boost confidence with each new deployment. In practice, a comprehensive suite of tests helps ensure that new features are introduced without causing regressions
Test engineers should exclusively focus on creating new tests. They should refrain from manual test execution, as it significantly prolongs the feedback loop for new features. Test execution is consistently automated by the CI/CD platform across different workflows and pipelines.
It is acceptable for a limited number of tests to be manually executed by individuals for smoke testing a release. However, this practice should be reserved for only a few select tests. All other primary test suites should be entirely automated.
Make Your Tests Fast
An extension of the previous point is the need for swift test execution. If test suites are to be seamlessly integrated into delivery pipelines, they must run quickly. Ideally, tests should complete within five minutes, and in no case exceed 15 minutes, which is comparable to the packaging and compilation times.
Swift test execution provides developers with assurance that the feature they’ve just committed is free from regressions and can be confidently advanced to the next stage in the workflow. A two-hour runtime is extremely detrimental to developers, as they cannot reasonably wait for such a prolonged period after committing a feature.
When the testing period is extensive, developers often shift their focus to other tasks and contexts. When test results do eventually arrive, addressing issues in a feature that is no longer their primary focus becomes more challenging.
Test data generation is another significant aspect where tests often consume a substantial amount of time. It is advisable to centralize and reuse test data creation code. If a test requires a lengthy setup phase, it might be attempting to test an excessive number of components or could benefit from mocking unrelated services.
To sum it up, test suites should ideally complete within a short timeframe (5-10 minutes), and any extensive tests that take hours to run should be subject to refactoring and redesign.
Each test automatically removes its own side effects
In general, you can categorize your unit tests into two additional categories (aside from unit/integration or slow and fast), which relates to their handling of side effects:
- Tests with zero side effects. These tests solely read information from external sources, refrain from making any modifications, and can be executed multiple times, or even in parallel, without any complications.
- Tests with side effects. These are the tests responsible for writing data to your database, interacting with external systems, executing output operations on your dependencies, and so forth.
Enabling each test to independently clean up its side-effects is a superior approach. This allows for parallel execution of all tests and enables individual test runs at any time, such as running a single test from the suite and subsequently rerunning it multiple times.
The ability to run tests concurrently is a fundamental requirement for utilizing dynamic test environments, as we will explore further in this guide.
Use Multiple Test Suites
Testing is not confined to a single step within a CI/CD pipeline; rather, it is an ongoing process that spans across all phases of the pipeline.
Maintaining a monolithic test suite covering everything can be unwieldy, leading developers to skip local testing. Ideally, developers should have the flexibility to choose and run any combination of test suites on their feature branches, enabling them to tailor their testing approach to their specific needs.
Create Test Environments On-demand
The conventional approach to testing an application just before it goes into production typically involves a single staging environment. However, relying solely on one staging environment has significant drawbacks. Developers are compelled to either test all their features simultaneously or join a queue to reserve the staging environment exclusively for their feature.
