I never ceased to wonder why universities don’t give equal weight to testing as to programming in computer science curriculum. Anyway, the best I can do here is to blog things that intrigue me!
The product I reorganized Test DB was an L4-L7 proxy
The test DB was organized neatly – but not in a black-box way. It had clients, servers and authentication folders at the top level.
The trouble was, a lot of tests were just UI test cases. UI had stabilized since ages. So experienced testers treated test DB as just another annoyance. They tested the box “how the user would use” and magically came up with bugs.
As a result, none knew exact quantum of test efforts. Also, the test coverage became a bit haphazard.
We took a strict black-box view and split tests by the “farthest” roles.
- “End user” was the farthest role. If anything involved an end-user to verify a configuration, the test had to fall in this folder. Authentication; Authorization; Accounting; Proxy Agent Installation; Connection setup, tearing-off and reestablishment etc. fell into this category
- “Administration and Appliance” was one step nearer. If an admin would like to care about the feature, the test had to fall in this folder. Fallback, government compliance, load balancing, compatibility with NMS/EMS etc. fell into this category
- “Support” was the third step. If a support person needed the feature, the test had to fall into this category. Logging, chatting, “God” mode etc. fell into this category
- “Legal and Commercial” was the nearest step. If the manufacturer needed the test, the test had to fall into this category. Licensing, packaging, live updates, copyright notices, logo and branding etc. fell into this category
The advantages were,
- Because end user scenario were few, number of test cases went down drastically. Any UI related test, whether on the client or on the server, was implicitly carried out while testing an end-user test case
- Because number of tests went down, associated scripts and related maintenance also went down
- Entire top level folders could be ruled out for regression testing. For example, if a patch didn’t change anything related to “Support” activity, snip goes a folder
The only seeming disadvantage was, the one-to-one mapping between a possible fault and a test case was lost.
Yes, as a puritan QA, I’d like to see “each bug is guaranteed not to return” during the regression testing cycle. But look at the fact that total number of test cases shrunk by 50%. If a user account couldn’t be created as an elementary test case, it could still not be created while making sure a VoIP call can be made!
Could the same be applied for testing a different solution – like a router? I think so.
The “end-user” aspect is actually testing just the fact that “the network is stable and bits flow through no matter what!”. That means, there is only one test – making sure the bits are going through. The “no matter what!” part is about fixing a topology and some combinatorial logic to generate permutations of as many disturbances as you care for.
The “administration and appliance” section is pretty much the same, with VRRP added
The “Support” and “Legal and Commercial” parts hardly change
What about a hierarchical application like banking or ticket reservation?
There instead of the four levels, more levels will be needed. The philosophy, “configure till this level and make sure it works or file a bug on the level it failed” doesn’t change.