Pairwise Testing: What is it and How to perform?

 Pairwise testing is a useful software testing technique. Let’s learn more about pairwise testing in the article below.

1. What is Pairwise Testing?

About Pairwise testing

Pairwise testing (all-pairs testing) is a software testing technique used to test all the possible combinations of various parameters involved. It is a permutation and combination-based method using a minimum set of test cases to reduce test execution time and cost, by focusing on interactions between pairs of parameters.

Many believe that pairwise testing not only reduces the test set size but also focuses testing on sources that have important bugs. For instance, in “Orthogonal Array Testing Strategy (OATS) Technique” J. M. Harrel has shown that:
– “Interactions and integrations are a major source of defects”
– “Most of these defects are not a result of complex interactions such as “When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.” Most of these defects arise from simple pair-wise interactions such as “When the font is Arial and the menus are on the right the tables don’t line up properly.”“

A series of studies by the National Institute of Standards and Technology (NIST) from 1999 to 2004, show that software bugs and failures are caused by one or two parameters, with progressively fewer by three or more, which means that combinatorial testing can provide more efficient fault detection than conventional methods.

We should evaluate the software being tested, figure out how inputs are combined to create outputs and how these operations could fail, and use an appropriate combinatorial testing strategy rather than blindly applying pairwise testing in all combinatorial testing scenarios.

Example

There is a simple scenario with three parameters:
– Operating System (O.S): Windows, MacOS.
– Browser: Chrome, Firefox.
– Connection type: Wifi, Ethernet.
With exhaustive testing, we have to test 2 x 2 x 2= 8 cases while using pairwise testing, we execute these 4 test cases instead.

Compared to Exhaustive testing, we don’t use the other 4 because we need to avoid duplicating the existing pairs:
– [Chrome – Windows – Ethernet] includes the pair [Chrome – Windows] selected in [Chrome – Windows – Wifi].
– [Chrome – macOS – Wifi] includes the pair [Chrome – Wifi] selected in [Chrome – Windows – Wifi].
– [Firefox – Windows – Wifi] includes the pair [Windows – Wifi] selected in [Chrome – Windows – Wifi].
– [Firefox – macOS – Ethernet] includes the pair [macOS – Ethernet] selected in [Chrome – macOS – Ethernet] and the pair [Firefox – Ethernet] selected in [Firefox – Windows – Ethernet].

These 4 test cases ensure all possibilities of each pair of parameters included: Browser – O.S, Browser – Connection type, O.S – Connection type. So it is still possible to detect bugs in selected cases. As the example above, if the case “Chrome – Windows – Wifi” passes, we don’t need to run cases that have pairs “Chrome – Windows”, “Windows – Wifi” and “Chrome – Wifi” because these pairs are all passed.

2. Benefits and Challenges of Pairwise testing

Benefits:
– Pairwise testing reduces the amount of time and cases needed to be executed (compared to exhaustive testing).
– This technique is easy to understand and implement, making it accessible to any level of testers.
– Without testing every possible combination, pairwise testing seeks to guarantee that all potential discrete combinations of inputs have been examined.
– With all pairs enforced, pairwise testing surely can detect defects between pairs of input parameters.

Challenges:
– Within the combination, it may be challenging to point out exactly which parameters or interactions cause the failure in that particular test case.
– When the system evolves and new parameters are added or existing ones are modified, covering all pairs and updating these test cases could be hard to ensure.
– On the other hand, by focusing on optimizing the coverage, some special combinations that lead to cases might be missed and leak some defects. So we need to emphasize some special cases to avoid missing major bugs.
– Besides, it is recommended to add additional tests to supplement the pairwise tests, which take more testing time and effort.

3. When to use Pairwise Testing?

When there are many input parameters in a software application and it is not feasible to create comprehensive test cases for every possible combination of parameters, Pairwise testing can be quite helpful. In many cases, most defects are caused by the interaction between pairs of parameters rather than testing each parameter. Pairwise testing ensures that every pair of values between different parameters is covered at least once.

In general, some scenarios like a large number of input combinations, lack of time and resources, and cross-browser testing are typical examples. To apply it wisely, it’s important to put Pairwise testing into a sensible perspective.

4. How to perform Pairwise Testing in Software Testing?

Pinpoint input parameters

Firstly, we need to point out that the right parameters are always the most important step of the whole process because if not, it will be wasting time and difficulty fixing test cases, which is the opposite of the convenience of using Pairwise testing.

Continue with the example, we have those parameters:
– Operating System (O.S): Windows, macOS.
– Browser: Chrome, Firefox.
– Connection type: Guest, User.

Create parameter tables

The first step of making the table starts with each pair of parameters.
Each pair (O.S, Browser), (O.S, Connection type), (Browser, Connection type) and their values can be grouped like this:

Generate test cases

We combine these pairs while making sure no pair gets left.

Moreover, clarify all expected outcomes for every test case to ensure these are as expected.

Run test cases

Afterward, execute those generated test cases and check again for any missed scenarios while testing. Besides that, record all results while executing, including any defects.

Analyze and report

Finally, analyze the results and provide insights or recommendations based on the analysis to enhance the quality or the improvement. The report should have some statements like defect classification, root cause analysis, test metrics such as test coverage, defect density, and trend analysis. They are useful for traceability, debugging, and regression testing.

5. Tips for keeping the process of Pairwise Testing efficiently

• More understanding of the domain, more suitable parameters selected: Before starting selected input parameters, deeply understand the domain and input variables, all possible values. Furthermore, this knowledge avoids irrelevant test cases that waste resources and time.

• Arrange test cases with priority and start with the most critical: Prioritize the parameters that have the highest impact on the system, are most likely to interact, or easily cause failures. This helps avoid missing important cases.

• Combine Pairwise Testing with other testing techniques: While Pairwise testing provides good coverage, it may not find some ad-hoc cases. Combine pairwise tests with boundary value analysis, exploratory testing, or some amount of exploratory/experience-based testing. Just need to ensure that combined testing techniques are suitable and follow the test target.

• Incorporate positive and negative testing: Besides testing in happy cases, we need to incorporate negative cases or invalid inputs in Pairwise testing to make sure the system handles errors correctly. 

• Limit the number of parameters to improve focus: By focusing on a smaller set of parameters, testers can manage the complexity of the test scenarios without overwhelming the testing process.

• Use Automated Tools to automate test case generation: Automated Tools are accessible to everyone these days. It can greatly expedite testing, lower manual error rates, and improve performance.

• Regular Review and Update Test Parameters: We should regularly review the test parameters to keep the testing up-to-date and updated in order to prepare for discovered problems, new features, and changing requirements.

• Monitor and Measure the Effectiveness: It is crucial to consistently monitor and assess the efficiency of Pairwise testing to identify strengths and weaknesses, refine strategies, and ensure that the testing process remains productive and beneficial.

6. Pairwise Testing tools support generating test cases

Using support tools to generate test cases helps utilize advanced algorithms to ensure comprehensive pairwise coverage and simplifies Pairwise testing.

PICT (Pairwise Independent Combinatorial Testing) is a powerful provided by Microsoft. It is a command-line tool generating pairwise test cases. PICT requires an input text file that lists the possible values for each parameter.
PICT allows for the definition of constraints to avoid generating invalid combinations. For example, we can modify the input file by adding constraints to prevent the combination of Safari and Windows. If we want to test some values more frequently, we can weight them by PICT.

ACTS (Advanced Combinatorial Testing System) is an open-source tool developed by NIST. ACTS is a Java-based application that is suitable for large and complex systems.
It supports Pairwise testing as well as higher-order combinations. This tool also provides different algorithms to generate test cases efficiently. Besides, it supports Constraint handling and Mixed strength allows to create and cover different parameter groups with different strengths.

Hexawise is a specialized tool that goes beyond basic Pairwise testing and is suitable for more complicated systems. It provides an intuitive interface and intelligent generation that optimizes for maximum coverage while minimizing the number of test cases.
Moreover, this tool also has advanced features like risk prioritization to prioritize cases based on risk factors, multi-way testing, weighted combinations or integration with test automation tools.

AllPairs is an open-source tool from the Satisfice. It helps generate test cases that cover all combinations of input parameters in pairs. This tool is a command-line tool and beneficial for straightforward Pairwise testing without the need for advanced configurations. AllPairs just needs an input file with parameter values. It is a lightweight tool with minimal configuration options and generates test cases quickly despite a large set of parameters and values.

7. Compare Equivalence Partitioning and Pairwise Testing

Comparison

Equivalence Partitioning widely used than Pairwise Testing

As a result, it is common for testers to use EP more than Pairwise testing. Here are some other reasons to explain the common:

• Simplicity and ease of understanding: EP focuses on values of a particular input (ranges or similar) so that testers easily apply it in frequent tests. Moreover, it’s a simpler process than to Pairwise testing, which requires more attention to the interaction of multiple variables.
• Common testing needs align with Equivalence Partitioning: Many practical testing scenarios frequently pay particular attention to validating individual inputs and making sure inputs work within valid and invalid ranges, which is exactly what EP addresses while Pairwise testing cannot combine all these values and other parameters although that can not guarantee the function works properly.
• Fewer test cases for simple scenarios: As the example before, EP shows the strength of reducing the number of test cases compared to Pairwise testing. Therefore, this makes EP less time-consuming in both creating and testing.
• Availability of tools and expertise: From the definition we discussed, Equivalence Partitioning shows that we can apply it without specialized tools while Pairwise testing needs some preparations. Furthermore, EP is a basic concept in black-box testing, making it more used than a complex technique like Pairwise testing, which may not be as widely understood or required.

In conclusion, EP concentrates on making the testing of particular input ranges simpler, whereas Pairwise testing aims for the best coverage when evaluating combinations of several input parameters. Use Pairwise testing when interactions between variables are important and need to be covered systematically, but EP is a helpful tool when working with certain input ranges or categories.

References

• Orthogonal Array Testing Strategy (OATS) Technique by Jeremy M. Harrell.

• Combinatorial Methods for Trust and Assurance