Test DB overhaul – Organizing black box test cases

I never ceased to wonder why universities don’t give equal weight to testing as to programming in computer science curriculum. Anyway, the best I can do here is to blog things that intrigue me!

Background:

The product I reorganized Test DB was an L4-L7 proxy

Before:

The test DB was organized neatly – but not in a black-box way. It had clients, servers and authentication folders at the top level.

The trouble was, a lot of tests were just UI test cases. UI had stabilized since ages. So experienced testers treated test DB as just another annoyance. They tested the box “how the user would use” and magically came up with bugs.

As a result, none knew exact quantum of test efforts. Also, the test coverage became a bit haphazard.

After:

We took a strict black-box view and split tests by the “farthest” roles.

  1. “End user” was the farthest role. If anything involved an end-user to verify a configuration, the test had to fall in this folder. Authentication; Authorization; Accounting; Proxy Agent Installation; Connection setup, tearing-off and reestablishment etc. fell into this category
  2. “Administration and Appliance” was one step nearer. If an admin would like to care about the feature, the test had to fall in this folder. Fallback, government compliance, load balancing, compatibility with NMS/EMS  etc. fell into this category
  3. “Support” was the third step. If a support person needed the feature, the test had to fall into this category. Logging, chatting, “God” mode etc. fell into this category
  4. “Legal and Commercial” was the nearest step. If the manufacturer needed the test, the test had to fall into this category. Licensing, packaging, live updates, copyright notices, logo and branding etc. fell into this category

The advantages were,

  • Because end user scenario were few, number of test cases went down drastically. Any UI related test, whether on the client or on the server, was implicitly carried out while testing an end-user test case
  • Because number of tests went down, associated scripts and related maintenance also went down
  • Entire top level folders could be ruled out for regression testing. For example, if a patch didn’t change anything related to “Support” activity, snip goes a folder

The only seeming disadvantage was, the one-to-one mapping between a possible fault and a test case was lost.

Yes, as a puritan QA, I’d like to see “each bug is guaranteed not to return” during the regression testing cycle. But look at the fact that total number of test cases shrunk by 50%. If a user account couldn’t be created as an elementary test case, it could still not be created while making sure a VoIP call can be made!

Could the same be applied for testing a different solution – like a router? I think so.

The “end-user” aspect is actually testing just the fact that “the network is stable and bits flow through no matter what!”. That means, there is only one test – making sure the bits are going through. The “no matter what!” part is about fixing a topology and some combinatorial logic to generate permutations of as many disturbances as you care for.

The “administration and appliance” section is pretty much the same, with VRRP added

The “Support” and “Legal and Commercial” parts hardly change

What about a hierarchical application like banking or ticket reservation?

There instead of the four levels, more levels will be needed. The philosophy, “configure till this level and make sure it works or file a bug on the level it failed” doesn’t change.

 

Advertisements

One comment on “Test DB overhaul – Organizing black box test cases

  1. pradheep says:

    Your thoughts are interesting. I like to discuss with you. Can you please contact through my site uni5.co

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s