Archive for the ‘Test Harness’ category

Waterfall manual test tools don’t work when the waterfall is broken

February 10, 2009

The Problem:

At Uplogix I’m now working with qatraq after we left testlink, and it is another tool that assumes you have a huge battery of good tests before you start testing.  You can assign them all to neat test scripts and assign the scripts and execute them. All of this falls apart when you realize the test is broken and you need to go fix it or you discover there are areas that are not tested. If you try to write one on the spot and add it to a script you end up loosing results for other tests.

Another problem that came up is how to deal with tests that have failed and rerunning them after the offending bugs have been fixed.

What I need to do

I should download qatraq, testlink and bugzilla/testopia at home and try to set up examples and see for myself if there is no workaround.  I noticed on,the pay-for-version of qatraq’s web site, that they make this claim: “The only Test Management tool based on IEEE Standards” which explains it not working with agile or exploratory testing.  What do IEEE/CMM shops do when they see cracks in their test plans?  Fix them in the next release in 9-18 months?  Spend a month or two on a patch?

Questions I have:

What, if any, test tracking tools deals with an environment where you have

  • Old test cases that may need to be reworked while testing
  • New discovered tests that a tester sees while testing
  • Exploratory or context driven testing where tests are developed and run at the same time without mounds of documentation that need to be maintained
  • Works with automated tests as well

What do you do if you are in an old-school IEEE, CMM, or Waterfall group (or even one that used the word process frequently) and want to migrate to a system that works better how do you deal with your old and new tests?

If the answer is ‘redo everything in the new technology’ you will probably fail.  Running the old system and new system separately for a transition period can be painful especially if the old system doesn’t die.

Test tools need to be more like revision control systems where you can migrate your assets from CVS to Subversion then later go to Git.

How do I track all the test equipment

August 29, 2007

I have ideas how to track it but I want to track it in a way that others could use the tracking tools also. The key is setting it up in a way that the test harness can see what is available, working then reserve it for the duration of the test. At Convio we had test systems that got a build and were set up a certain way that was needed to run most of the tests. If we had automated tests that would have been the target, the parameters you feed in are: host name, db credentials and possibly what browser platform to use.

At Uplogix I have 2 pieces of equipment to test: and Envoy that you ssh into to control various devices and an EMS that has a web front end that controls many Envoys. The configuration is complicated by what devices are physically connected to the Envoys (routers, switches, GPS units, Unix machines or simulators for these devices) and how they get connected to the EMS.

I want to solve this in a generic way. Defining resource types (Envoy, EMS, Serial device, power controller,…) instances of those resources and how they are connected. Then I can write scripts to make sure the devices are functioning (they break a lot with what we do to them) configure them when new builds show up, and allow test cases to find what it wants then reserve it. That last part is the most interesting.

So I may have 7 Envoys of two different flavors, with 4 routers connected to them, 3 of them are connected to a power controller. A test case that needs to break into a router by power cycling the router and catching the prompt and sending a new config to it would have to look for this:

(Envoy: type=’TU301′, name=$envoy, reserved=false) and (2600: envoy=$envoy, power=$power, port=$port, reserved=false)

So prolog or an SQL inner join, it would match the name of the TU301 with the 2600’s envoy so it knew they are wired together (and not reserved by another test or person) and give you the name of the devices, the port id of the Envoy, and the power port id to pass to the test case.

In an SQL transaction it would have to reserve the unreserved equipment or roll back and wait or try a different set of gear. Once the tests are complete it would unreserve the 2 pieces of equipment.

When that TU301 test is done we now look for ‘Spitfire’ instead and run the test again with the new parameters. If we don’t find matching equipment we flag it as skipped due to lack of resources or busy resources for a person to fix.

For bonus points I can run the sanity tests on both devices after the test and if the router is in ROMMON mode or unresponsive I know what tests caused the problem and take the device out of service. Possibly disable the test too.

So now I have tests running every night

August 28, 2007

I have tests running every night, great. Now I need to monitor the results.

The tests are written in Perl and to test a CLI. The tests are not super important or contain any measurable coverage but allow me to see a problem right off the bat that needs to be solved now. I now have to go look at a file generated and see what the deltas are. Here is how I got to this point.

  • Create Perl libraries to figure out what build is on a machine and load a different one
  • Make an object that encapsulates all the Expect code to communicate with the device I am testing.
  • Generate tests using the object with Perl’s Test::More module.
  • Write a program that looks at Cruise Control build machine finds the latest build use the library to load this onto the test machine then run the tests.

So, here is the problem now, I am looking at a file every morning to see what the results are. I see this output:

ok 1 - timeout set to 120 minutes
ok 2 - timeout set to 5 minutes
ok 3 - timeout set to 101 minutes
ok 11 - timeout 5 minutes after disconnect
ok 12 - timeout set to 120 minutes

So I now want to move this to my database and have to program e-mail when there is a delta. I can look at the results with some CRUD scaffolding. Later, there will be web testing that will use Watir and be in Ruby.

How do I do this?

Write something like Test::More in Perl and Ruby that connects to a database?

Or extened Test::More to do this. What about Ruby? Extend RSpec to do this?

I like the power you get from tools like RSpec and Test::More but do they work well outside of a unit test environment?

Nobody cares if your automated test passes or fails.

August 7, 2007

Years ago when I worked at Convex Computer Corp we used something that is now called Convex Integrated Test Environment or CITE. While running a set of tests you would see things like PASSED which made you feel good or FAILED – RC 1 expected 0 which made you feel less good. I tested the compilers, libraries and similar tools which shipped with very few known errors, but we did have them. The errors would involve things like, should have vectorized this loop or got an IEEE NaN instead of Infinity but I soon learned that people didn’t care about the failures. That surprised me. We had a tool that post processed the results and compared them against what was expected if we expected this bizarre Fortran expression with 10000 sub expressions to produce a compiler error and it is still giving the same error, that is fine. When the tests were done being run people would be interested after the post processor looked at the results. Then and only then did people take action. Qualification of a single compiler took about 2 weeks with nearly 100% test automation with tests running on millions of dollars of hardware all over the company. Waiting 8 hours for the real test results was tough for a new collage graduate with a whole department anxiously waiting results. This was a great system for the day.

When I left the scientific computing world behind for things built with these compilers and libraries I found the tolerance for known problems to be much higher. Leading the testing effort for a startup later I decided to build in the expected and unexpected. Now when the tests run you would see: EXPECTED – PASS and feel good but still feel good when you saw EXPECTED – FAIL RC=2 expected 0 (bug 123). This was good as I was now testing medical software (that they still seem to be selling today at Quovadx, I hope my tests are still running) and using less than $50,000 of equipment. The tests had only 24 hours to be run and the software was arguably more complex as an RDB was employed, multiple config files, threads in the early days of threads and eventually multiple platforms including the dreaded Windows NT.

It turned out that building in a system for Expected/Unexpected was a good design when multiple platforms came along with different results. The system that captures a signature for a test failure was remembering the associated but that went with it now had another parameter, platform, added to it. With many sets of expected results you could now scan the old ones to look for regressions or one platform acting like another. This was good.

Now I am at Uplogix and I am faced with the same problem. However, the UI is far more complicated. I not longer use canned Fortran source or a Perl script to generate config files and run executables on them. I have a Cisco IOSish CLI on a hardware appliance that I have chosen to communicate with Perl‘s library. I have also done a tiny amount of web automation with Watir on our web product.

What do I do now? The Web management for our Envoy machines are tied very closely. I’m a huge fan of Perl but tools like Samie are not at the same level of Watir and Ruby’s expect is not as powerful as the TCL or Perl versions. It is time to build the test harness that binds all of this together. We have multiple platforms again in the form of different browsers. Tests now consist of many machines so remote communication will be needed.