About management systems and test automation framework

(These are my views. It has nothing to do with my current or past employers. The narration here doesn’t infer my experiences with or existing practices of my current or past employers.)

Eventually every networking product evolves a management software either in form of an element management system (EMS) or network management system (NMS). Similar approaches are also taken by storage industry, bio-medical device industry and so on. Essentially, an embedded software needs some management software.

It is also true that in this combination, larger portion of intellectual property lies within the embedded product. It is usually the first one to get developed, sold and later for margin extension management software is developed.

If it were not for margin extension, most companies would line up with OpenNMS kind of a solution. A proprietary management system also allows a vendor lockup.

However, there also exists a keen, silent awareness that a management software is a secondary product. Less resources are given for its engineering and testing. If a management software product is developed and tested fully, it could be really a USP – but it seldom happens.

On the other hand, such step-motherly treatment is also given to the core embedded product’s test automation framework. Actually it is even worse to be in charge of a test automation framework. The work demands maturity of an architect but is usually treated as even less important than testing or test automation.

If you take a step back and look, both pieces of software are doing the same thing – using an application software to control the core embedded software.

A management system typically cares about FCAPS (Fault, Configuration, Accounting, Performance and Security). In many commercial systems it allows scheduled tasks.

A test automation framework also does the same! It should have a scheduling mechanism, configuration mechanism, fault monitoring to raise errors, accounting and performance measurement modules for system testing and of course, security framework – to protect itself and also in form of penetration testing.

That means a large overlap exists in the requirement sphere of the two. Ideally, common libraries and even common front-ends should be written for the two. When it runs in customer mode, it should act towards management; when it runs in testing mode, it should run towards testing automation.

Not only testing resources are saved that way, we can save a lot of testing efforts too. Say if 60% code of the management system is common with the test automation framework, during test automation runs, this 60% overlap will be hammered like anything!

Then the question is, why is this not a practice? The answers are surprisingly not technical.

  1. As I mention earlier, management systems aren’t usually implemented later than the embedded software.  Often their development is later than even the test automation framework – at best they are developed in parallel with the test automation framework. Program management for absorbing both of them is hard
  2. Also, as I mentioned, there 60% overlap in functionality – but there is also a gap of 40% each. That means, you have to spend 140% to get both. Most of the time, senior management (rightly) is unwilling to risk so much at the same point of time – when one of them is a secondary software and the other isn’t even going to fetch any top line!
  3. Here is the dirty secrete of Hi-tech industry (or as it is called IT industry in India). It has a caste system. Hardware engineering is held at the highest esteem, followed by embedded software, followed by application software, followed by test automation, followed by testing, followed by support. This hierarchy reflects in choice of tools – like hardware is tested in C++ – and applications may be written in C++; embedded software is written in C/C++ and often programmers feel offended if they are asked to try higher level language – and are tested using either scripting languages or human languages (manpower). In this hierarchy, a development manager for a management system, a Java guy, usually finds it extremely offensive to agree that his/her problem space has so much in common with “something as mundane as a test automation framework”, which is “nothing but a bunch of scripts cobbled together”. Every hierarchy, every caste system doesn’t exist on the feeling of inferiority of  “I am sitting higher than y but I am sitting lower than x”. It works on the feeling of superiority of “I am sitting lower than x but I am sitting higher than y”. The world needs everyone with fairly the same importance – but human pride makes such hierarchies sub-optimal yet rigid. Writing a common software for the use of high-caste developers and low-caste testing usually becomes unthinkable, unspeakable or at least impractical.

As I mention in one of my jokes, sadly, technical problems are often the simplest.

Q: What is harder than to colonize Mars?

A: To get the budget approval for it!


Black box testing automation and the myth of productivity improvement

[Look, who is talking? I have spent nearly 12 years in automation, automation framework development AND testing and attribute my career success to my contribution to automation.]

There is a general belief in software testing industry that test automation enhances productivity of the testing team.

While this is true to some extent, it isn’t true all the time. Automation is no silver bullet.

Especially in black-box testing, automation may not be needed or affordable after all.

Here are some reasons – and I would like you to give me more reasons.

  1. Maximally covered black box test automation gets nearly as complex as the product itself. This is because if any feature in the product that is not accessible to the end user or administrator, why is it there in the product in the first place?
  2. As the product changes the UI, its black box test automation also need to change. Lemma of this is, if your horizon is longer than a few major releases, black box test automation will need exactly the code branching and other software engineering sophistication as your product
  3. If your testers could code, why would you not use them for coding and beating the competition?
  4. Most test databases have 80% test cases as “configure and verify it works”. As the product matures, necessary regression tests are just a handful of use case scenario. If it is a router, [test that] it must route the packets – if it fails, troubleshoot why it fails — but core BGP doesn’t have to go through configuration tests all the time. If it is a banking application, customer name display module doesn’t change once it is written and verified. We end up spending money in automating ALL THOSE once important later trivial  “configure and verify it works” tests, maintaining them and running them

In summary, often through black box testing automation, we are

  • undertaking a complex task
  • with increasing maintenance cost
  • probably misusing manpower and
  • with diminishing returns on investment

Only the last of the challenges mentioned above can be cured by periodically cleaning up test database.

What do you say?

The purpose of automation

Of late I have been wondering what is the purpose of automation?

  • Is it improved Quality by elimination of erring manpower and delegating the work to machines?
  • Is it saving Time by eliminating coffee break taking manpower?
  • Is it bringing down Cost by eliminating unionizing manpower?
  • Is it elimination of manpower for the sake of elimination?

The questions sound too theoretical. However, depending on how you answer, approaches to automation may change!

For example,

  • if you put Quality over Time and Money, you may end up designing a 100,000 dollars hammer. You may end up so complex automation that by itself may require testing. Just think of automated fab! It takes years to build and years to test before it goes into production.
  • if you put Time over Quality and Money, you may end up spending fortunes.
  • if you put Money over Time and probably Quality, outsourcing may be easier than automation.

I have designed automation systems that each optimize one of {Quality, Time, Money} at the cost of the other one or two. The management (of any company I was working with) often changed the need half-way – and then wondered why automation efforts are not paying off.

As a designer, it used to tick me off a lot.

Only when I became an experienced manager, I realized that it is not a simple myopia – it is a case of astigmatism! Priorities do pose themselves in a jumbled manner! For a year, you are competing hard in a growing market – and Time reigns supreme. Next year, you see that the market has screwed up and Money becomes the priority at yet another time, your best engineer has left the job and automation better ascertains Quality, even if the release can go a week later.

I have also realized that (in test automation at least) it is possible to come up with a system that optimizes all the three parameters. However, it requires laying down foundations of the automation framework very carefully. More on that later 🙂

When is an n layered network stable? (pseudo-mathematical definition)

  • An n layer network is a strictly ordered set of digraphs G1,G2,…,Gn such that
    • Vn is a subset of Vn-1 is a subset of … V1; where Vi is a set of vertices of Gi
    • For m > 1, a path Pm(i,j) on vertices i, j in Vm exists only if Pm-1(i,j) on Vm-1 exists
  • An n layer network is stable if and only if
    • for all i,j in Vn, all paths Pn(i,j) are stable
  • A path Pm(i,j) is stable if and only if for all q<=m, all Pq(i,j) are stable

Testing as Proof

Often I think that the ideal language for implementation of test automation control logic is Prolog!
Now, you’d say only professors talk of Prolog.
Let us see what I mean here.
Testing is after all proving that the system works or not.
System works if component A works and Component B works and …
This is very similar to Prolog programming, isn’t it?
That means test automation scripts must be controlled by Prolog kind of an engine.
I know there are more details to be discussed. I am just sowing a seed here.