Black box testing automation and the myth of productivity improvement

[Look, who is talking? I have spent nearly 12 years in automation, automation framework development AND testing and attribute my career success to my contribution to automation.]

There is a general belief in software testing industry that test automation enhances productivity of the testing team.

While this is true to some extent, it isn’t true all the time. Automation is no silver bullet.

Especially in black-box testing, automation may not be needed or affordable after all.

Here are some reasons – and I would like you to give me more reasons.

  1. Maximally covered black box test automation gets nearly as complex as the product itself. This is because if any feature in the product that is not accessible to the end user or administrator, why is it there in the product in the first place?
  2. As the product changes the UI, its black box test automation also need to change. Lemma of this is, if your horizon is longer than a few major releases, black box test automation will need exactly the code branching and other software engineering sophistication as your product
  3. If your testers could code, why would you not use them for coding and beating the competition?
  4. Most test databases have 80% test cases as “configure and verify it works”. As the product matures, necessary regression tests are just a handful of use case scenario. If it is a router, [test that] it must route the packets – if it fails, troubleshoot why it fails — but core BGP doesn’t have to go through configuration tests all the time. If it is a banking application, customer name display module doesn’t change once it is written and verified. We end up spending money in automating ALL THOSE once important later trivial  “configure and verify it works” tests, maintaining them and running them

In summary, often through black box testing automation, we are

  • undertaking a complex task
  • with increasing maintenance cost
  • probably misusing manpower and
  • with diminishing returns on investment

Only the last of the challenges mentioned above can be cured by periodically cleaning up test database.

What do you say?


2 comments on “Black box testing automation and the myth of productivity improvement

  1. James Whittaker in How Google Tests Software has (among many others) mentioned a key issue about Automation Planning:

    “Over-investing in end-to-end automation often ties you to a product’s specific design”

    This is true. A QAE’s time is limited and spared thinly. It is good idea to start thinking of automation since the very early stage. However, over-investing in test automation is not particularly useful until the entire product is built and in the stable form. Because by then (hopefully) there won’t be any system design change.

  2. […] This dream is never achieved because of reasons I mentioned in one of my previous posts. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s