Nobody cares if your automated test passes or fails.

Years ago when I worked at Convex Computer Corp we used something that is now called Convex Integrated Test Environment or CITE. While running a set of tests you would see things like PASSED which made you feel good or FAILED – RC 1 expected 0 which made you feel less good. I tested the compilers, libraries and similar tools which shipped with very few known errors, but we did have them. The errors would involve things like, should have vectorized this loop or got an IEEE NaN instead of Infinity but I soon learned that people didn’t care about the failures. That surprised me. We had a tool that post processed the results and compared them against what was expected if we expected this bizarre Fortran expression with 10000 sub expressions to produce a compiler error and it is still giving the same error, that is fine. When the tests were done being run people would be interested after the post processor looked at the results. Then and only then did people take action. Qualification of a single compiler took about 2 weeks with nearly 100% test automation with tests running on millions of dollars of hardware all over the company. Waiting 8 hours for the real test results was tough for a new collage graduate with a whole department anxiously waiting results. This was a great system for the day.

When I left the scientific computing world behind for things built with these compilers and libraries I found the tolerance for known problems to be much higher. Leading the testing effort for a startup later I decided to build in the expected and unexpected. Now when the tests run you would see: EXPECTED – PASS and feel good but still feel good when you saw EXPECTED – FAIL RC=2 expected 0 (bug 123). This was good as I was now testing medical software (that they still seem to be selling today at Quovadx, I hope my tests are still running) and using less than $50,000 of equipment. The tests had only 24 hours to be run and the software was arguably more complex as an RDB was employed, multiple config files, threads in the early days of threads and eventually multiple platforms including the dreaded Windows NT.

It turned out that building in a system for Expected/Unexpected was a good design when multiple platforms came along with different results. The system that captures a signature for a test failure was remembering the associated but that went with it now had another parameter, platform, added to it. With many sets of expected results you could now scan the old ones to look for regressions or one platform acting like another. This was good.

Now I am at Uplogix and I am faced with the same problem. However, the UI is far more complicated. I not longer use canned Fortran source or a Perl script to generate config files and run executables on them. I have a Cisco IOSish CLI on a hardware appliance that I have chosen to communicate with Perl‘s Expect.pm library. I have also done a tiny amount of web automation with Watir on our web product.

What do I do now? The Web management for our Envoy machines are tied very closely. I’m a huge fan of Perl but tools like Samie are not at the same level of Watir and Ruby’s expect is not as powerful as the TCL or Perl versions. It is time to build the test harness that binds all of this together. We have multiple platforms again in the form of different browsers. Tests now consist of many machines so remote communication will be needed.

Advertisements
Explore posts in the same categories: Test Harness

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: