Take for example testing of a router where we list features in columns of test areas:
In the beginning of a product like this, most interest will be in making sure each of the protocols work. Stand alone testing of the cells will be more than enough.
Once most low hanging (and highly damaging) bugs that could be found through testing cells of the matrix alone, are weeded out of the software, QA progresses by making “experts” of testing going column-wise – L1 expert, L2 expert, L3 expert etc. This leads to more experienced testers and technically great bugs (like route redistribution having memory leaks). QA also arranges its test plans by test areas and features.
At this stage, only a section of bugs seen by a customer is eliminated and the complaint “QA isn’t testing what the customer ways” continues.
That is because the customer doesn’t use the product by columns. A typical customer environment is a vector selected one member from each column (at most). A product is likely to break at any vector.
As you can see, overall testing of the matrix above would require testing 4*4*5 = 80 environments. In reasonable products the actual number may be in 10,000’s.
Testing a feature in presence of many possible combinations of environment is a well-known QA problem.
Various approaches have been suggested. There have been combinatorial engines and test pairs and so on to help QA optimize this multi-dimensional problem.
The approach I discuss here is yet another algorithm to be followed by semi-automation. Just let me know your thoughts about it.
Let us define our terms once again:
- Feature: A feature in the product [I know that isn’t a very great definition.]
- Area: A set of related features
- Product: A set of Areas
- Environment: A vector (or a sub-vector) of Product, with each component coming from an Area
- Environment for a Feature: A maximal length sub-vector of the Product without the Area of Feature
Please understand once again that QA expertise and test plans are structured by Area (a column in the matrix). The best way to test will be to test every test of every Feature against every “Environment for a Feature”.
This approach is simply uneconomical. So, what is the next best approach?
Before coming to that, we need to understand how test cases are typically structured. In a feature, typically test cases have some overlap of steps – like configuration or authentication etc or overlapping phenomena like exchange of Hello packets, establishment of neighborhood etc.
That means, there is a significant redundancy of tests from white-box point of view.
This redundancy can be exploited to assure that the product may stand reasonably well in diverse environments. As we discussed earlier, such an environment is a vector of that matrix, which in turn is a sub-vector plus a test condition.
Understanding so much brings us to a workable solution for testing more “customer-like” without incurring too much cost.
The key understanding from the above discussion is that Environment for a Feature can be changed independently of the test case (or the Feature or the Area).
That means if the tester can signal an “environment controller” at the end of a test case, the controller may change the DUT/SUT to another Environment for the Feature. After that change is done, the tester just continues testing the system to the next test case – till all test cases end.
Because it is less likely that number of test cases are a factor (or multiple) of number of sub-vectors, within a few test cycles reasonable amount of test steps will be tested across reasonably diverse environmental sub-vectors.
As a strategy of testing the product, QA can assure its internal customers that over a few releases, most interesting bugs that can be found economically will be found.
What are the downsides of this approach? For sure, the Environment for a Feature must not have any configurations about the Feature under test – or even the Area. That means the tester will have to always configure the Feature before going to the next test. If you are working on a Feature that takes VERY long to converge, you are out of luck. [VPLS in networking comes across as an example.]
Since most products don’t involve long signaling delays, let me focus on the optimization of this testing.
How can we find maximum number of bugs in a given feature (or the entire column) related to environmental changes within minimum number of steps?
The answer is obvious to the readers of this blog – by taking the sub-vectors in anti-sorted way!