测试用例生成工具 —— PICT(Pairwise Independent Combinatorial Testing tool)
t-wise testing is t-wise testing with a thoroughness of t.
The term all-pairs testing is simply equivalent to pairwise testing.
Mixed-strength testing is to mix, for example, 2-wise and 3-wise testing in the same round of testing.
Combinatorial testing is a more general term that includes pairwise testing but also other testing techniques.
Combinatorial testing > Pairwise testing > Orthogonal array testing
Pairwise testing fits nicely into the testing phase of software or hardware development. It works well with agile development practices.
Pairwise testing is usually not fully automatic. Pairwise testing usually requires some manual work to describe the thing that is to be tested.
Usually a pairwise testing tool then takes this description (or model) and generates the pairwise tests automatically.
Pairwise testing is more focused on putting your system into varied situations.
Having a domain expert make tests for a software or hardware system is valuable and highly recommended.
Pairwise testing is not a replacement for this kind of testing but complementary to it.
Pairwise tests acts in a similar way, it explores the different paths of interaction.
From a user’s point of view, pairwise testing is light-weight and simple to apply to even the larger cases.
From the pairwise tool developer’s point of view, however, pairwise testing involves some heavy-weight computations. This was until the last few years a bottleneck in the application of pairwise testing to the larger real-life cases, but in recent years effective algorithms for pairwise testing has been invented and are now available in commercial products.>>> Some papers on the computations.
The effectiveness of pairwise testing is neither hypothetical nor only academic. The effectiveness of pairwise testing has been studied by looking at existing large-scale software or hardware and their histories of problems and bugs. As the thoroughness of pairwise testing increases, it has been found that increasing amounts of bugs can be found. I one large study, it was found that for a thoroughness of 1, 2 and 3, the percentage of bugs that might be found was 50, 75 and 90%, respectively. This study found no bugs that required more than a thoroughness of 6, a result that has stood the test of time. This is the reason why a thoroughness of 6 is usually the maximum considered.
As the thoroughness of pairwise testing increases, so does the effort required to do the tests. A thoroughness of 2, i.e. 2-wise or pairwise, has a very good tradeoff between thoroughness and effort. A thoroughness of 3 is a good for more important rounds of testing, for example, for later stages of development.
For testing highly critical software of hardware, pairwise testing is a good start, but other quality assurance techniques should be used in addition.
A pairwise test suite might contain, for example, 30 test cases. The accumulated coverage after each test case has been added will typically form a curve like this:
As we can see in this example, the first 12 tests include more than 95% of the pairs. So, the last 18 tests cover less than 5%.
In ordinary pairwise test generation, the pairs included (pseudo-)randomly, meaning that any faulty pair is most likely to be included early in the test suite. Thus, a partial test suite of, for example, 95% can be expected to detect 95% of the bugs that a complete pairwise test suite would find.
Pairwiser(Web-based, free & commercial)
Pairwise testing is not well suited for a testing problem with a few parameters with relatively many combinations. As we will see in other examples, pairwise testing performs very well when we have more parameters, which the most interesting, real industrial cases surely tend to have!
Pairwise testing was combined with a kind of equivelence class selection technique to reduce the number of values to include for generating the pairwise tests. This is a quite common technique to combine with pairwise testing.
- The parameters: primary/second/...
- The constrains: always together/if..then..
- The required test cases.