[Look, who is talking? I have spent nearly 12 years in automation, automation framework development AND testing and attribute my career success to my contribution to automation.]
There is a general belief in software testing industry that test automation enhances productivity of the testing team.
While this is true to some extent, it isn’t true all the time. Automation is no silver bullet.
Especially in black-box testing, automation may not be needed or affordable after all.
Here are some reasons – and I would like you to give me more reasons.
- Maximally covered black box test automation gets nearly as complex as the product itself. This is because if any feature in the product that is not accessible to the end user or administrator, why is it there in the product in the first place?
- As the product changes the UI, its black box test automation also need to change. Lemma of this is, if your horizon is longer than a few major releases, black box test automation will need exactly the code branching and other software engineering sophistication as your product
- If your testers could code, why would you not use them for coding and beating the competition?
- Most test databases have 80% test cases as “configure and verify it works”. As the product matures, necessary regression tests are just a handful of use case scenario. If it is a router, [test that] it must route the packets – if it fails, troubleshoot why it fails — but core BGP doesn’t have to go through configuration tests all the time. If it is a banking application, customer name display module doesn’t change once it is written and verified. We end up spending money in automating ALL THOSE once important later trivial “configure and verify it works” tests, maintaining them and running them
In summary, often through black box testing automation, we are
- undertaking a complex task
- with increasing maintenance cost
- probably misusing manpower and
- with diminishing returns on investment
Only the last of the challenges mentioned above can be cured by periodically cleaning up test database.
What do you say?