I’m now a mobile tester

Posted December 24, 2013 by Brent
Categories: Uncategorized

I’ve decided to take a job writing automated tests for HomeAway’s mobile applications.  So I’ve gone from using Watir and Watirmark to something new.  What I have to test  now can be broken down to these parts

  1. An iOS native application
  2. An Android native application
  3. A set of APIs that the native application uses for data CRUD
  4. An app for owners that is written using Phonegap

I started about 6 months ago and have been learning the business and the applications as well as looking at ways of testing that takes what I have seen work in the past.  When I first started writing proof of concept tests I used Appium Selenium and Ruby for testing 1 and 2 above.  The first big discovery is that I can’t use IDs to get at the elements.  I have to use the accessibilityIdentifier on iOS and the contentDescription on Android.  While the applications are developed by two different sets of developers and are not the same I was able to do some useful testing with the exact same test code.  To do this I had to get both dev environments up and running on my machine and put in the same accessibility strings in the two systems.  I imagine there will be places where UI code will look like

if platform == 'iOS'
do this
do that

I have not had to do that yet.

For testing the APIs I am using net/https and json.  This testing was easy to get going and is in TeamCity running after the API it is testing builds.

The Phonegap application is interesting.  I spent a couple of days trying to use Appium or variations of Appium to get at the application bit it reminded me of testing flash applications embedded in HTML  So, I am running the application as a web application on the local machine and using Selenium to drive a web browser and that is working now.

All of this code uses a Watermark-like MVC approach to test their parts with the exception of the API tests, in that case the View code is just the JSON and the controller just calls out to JSON instead of a normal view.  The JSON controller is subclassed from the normal controller and probably needs a little work.



rspec and instance variables

Posted April 4, 2011 by Brent
Categories: General Test

I was working on watir koans at Watirday 2011 and ran into a snag.  I had assumed I did something careless and needed to look at it when I could focus better.  Later that night I noticed that night I realized I just didn’t understand the scoping rules.  I created a new watir-webdriver object in the before(:all) block and used it in the examples for the example group.  The problem started when I didn’t want to replicate finding the form in later examples so I created a new @f class variable in an example to hold the form but it was nil in the next example.

I was lucky to have an experienced ruby tester, Marek J, there to explain what had happened.  I heard about class variables, classes within classes and copies of variable appearing in the examples.  If you change the variable in one example you will see it go back to the original value in the next example.  This is rspec helping isolate the examples.

Simple right?  No. there is an exception you need to know right away.  If your variable is a reference to something, in the heap, then modifications to that data will be referenced through the class variable.

Ok, so simple variables will be reset but references to complex variables, like a hash, will change keep state between examples?  Well, yes and no, the final example below shows that changing the instance variable to point to a different hash will be reset between examples just like the simple data types.  You kinda have to know about pointer or references to get it.  This happens in Perl too so I understood.  I googled a little and didn’t see it explained so I wrote this blog entry.

The root of what I’m describing is instance variables in classes and subclasses. This is relevant to rspec because:

  • the describe block is magic that makes a class
  • the it block is magic that makes sub class of the describe class
  • the before and after blocks are not in a sub class they are part of the describe class

Because the tester doesn’t see the module or class in their code they may not know what is happening under the covers.  However, they do need to know the behavior or they may learn to just use globals.

require 'rubygems'
require 'rspec'

describe "rspec for simple variables" do
  before(:all) do
    $global   = 'sg'
    @@class   = 'sc'
    @instance = 'si'

 it "should see all simple before variables" do
   $global.should   == 'sg'
   @@class.should   == 'sc'
   @instance.should == 'si'

 it "should change all the simple before variables" do
   $global   = 'new global value'
   @@class   = 'new class value'
   @instance = 'new instance value'
   #modifying a copy of the one defined in the before block

   $global.should   == 'new global value'
   @@class.should   == 'new class value'
   @instance.should == 'new instance value'

  it "should verify changed simple variables in this example" do
    $global.should   == 'new global value'
    @@class.should   == 'new class value'
    @instance.should == 'si'
   # Surprise, @instance.should is still 'si' from the begin block

describe "instance references can be changed" do
  before(:all) do
    @instance_hash = {:i_live => "heap memory"}

  it "should see the heap variable" do
    @instance_hash.should == {:i_live => "heap memory"}

  it "should modify the heap variable" do
    @instance_hash[:i_live] = "modified hash"

  it "should see the instance new value via the instance var" do
    @instance_hash[:i_live].should == "modified hash"
    # What just happened? The string was not reset between
    # examples. It points back the same memory but the contents
    # of the memory have changed.

  it "should set the class hash to a different hash" do
    @instance_hash = {:i_live => 'new hash'}
    # This hash will be lost forever when @instance_hash
    # goes out of scope when this example finishes.

  it "should still have the modified hash from the before block" do
    @instance_hash[:i_live].should == "modified hash"
    # The previous example changed the reference of hash so it
    # was restored to point to the one from the begin block.

Sorry about the bad formatting, I don’t know what WordPress knobs to turn to make this page wider without paying money.

Waterfall manual test tools don’t work when the waterfall is broken

Posted February 10, 2009 by Brent
Categories: Test Harness

The Problem:

At Uplogix I’m now working with qatraq after we left testlink, and it is another tool that assumes you have a huge battery of good tests before you start testing.  You can assign them all to neat test scripts and assign the scripts and execute them. All of this falls apart when you realize the test is broken and you need to go fix it or you discover there are areas that are not tested. If you try to write one on the spot and add it to a script you end up loosing results for other tests.

Another problem that came up is how to deal with tests that have failed and rerunning them after the offending bugs have been fixed.

What I need to do

I should download qatraq, testlink and bugzilla/testopia at home and try to set up examples and see for myself if there is no workaround.  I noticed on http://www.testmanagement.com/,the pay-for-version of qatraq’s web site, that they make this claim: “The only Test Management tool based on IEEE Standards” which explains it not working with agile or exploratory testing.  What do IEEE/CMM shops do when they see cracks in their test plans?  Fix them in the next release in 9-18 months?  Spend a month or two on a patch?

Questions I have:

What, if any, test tracking tools deals with an environment where you have

  • Old test cases that may need to be reworked while testing
  • New discovered tests that a tester sees while testing
  • Exploratory or context driven testing where tests are developed and run at the same time without mounds of documentation that need to be maintained
  • Works with automated tests as well

What do you do if you are in an old-school IEEE, CMM, or Waterfall group (or even one that used the word process frequently) and want to migrate to a system that works better how do you deal with your old and new tests?

If the answer is ‘redo everything in the new technology’ you will probably fail.  Running the old system and new system separately for a transition period can be painful especially if the old system doesn’t die.

Test tools need to be more like revision control systems where you can migrate your assets from CVS to Subversion then later go to Git.

Where did IBM blow all that smoke?

Posted September 26, 2007 by Brent
Categories: General Test

I am waaaaaaay to busy to be distracted today. There is a release going out on Friday and like usual there is a lot of work to do. So when I got a Dr. Dobb’s e-mail with something that looked somewhat test oriented that pointed me to IBM I went. This paper seemed like it took testing one step farther than record and playback testing as it would try to figure which ‘test case’ you were executing and offer to finish it out for you. The more I read the more I felt like I was reading sci-fi. Then I got to this on page 4 of 4: “Although these are currently unimplemented, we will give a brief overview of some of the possible approaches” I now know where the smoke was getting blown. Wow, good luck with that! I feel like someone from the testing community should say, hey we are working on these problems… and they are real hard to solve.

Should there be a Watir users group in your town?

Posted September 20, 2007 by Brent
Categories: General Test

I think it is time for Austin to get a Watir users group.  Bret Pettichord checked the interest of this about a year ago, the response was good.  It may be even better to start one now.

In the last year I have seen some great advances of what to do with Watir to help people really take this technology and start writing practical tests very quickly.  Last week Hugh McGowan and Rafael Torres showed off a flexible data driven test system that made each row or column of a worksheet into a set of Rspec tests.  This was open source software on ruby forge.  Then shortly after that Mark Anderson used this to write a data drive test to hit google given search terms to verify the expected top url.  This code should be on the same ruby forge site now.  The demo was quick and impressive.
With this advances I saw someone ask a question on the Austin Watir mailing list how to install Watir on their system.  There was a quick answer.  The Wati-general mailing list is very busy these day with advanced and newb questions.

Now seems to be a good time to get an organized Watir group set up in Austin.  I would also like the idea of exporting our presentations (an importing as well) to user groups in other cities to get things moving that way as well.

I’m thinking a meeting would be called when we have a venue and 2 presenters and might go like this:

5:30-6:15 Early arrival, people can bring questions to see if a more experienced user can help them.  Possibly as advanced as installing Ruby/Watir/Test tools on the machine or a recent user-group presentation demo.

6:15-6:30 Regular arrival, people can announce that they are looking for Watir users and testers or are looking for work in the field.

6:30-7:00 Watir report – Someone reports experience with the Watir or related tools at work.

  •   This was the problem.
  •   We did this
  •   This worked/this didn’t

7 :10-7:50 Demo – Showing off a new tool, module or library

  • See Watir in action
  • Learn about coding techniques or a tools
  • Later you can down-load and try out something

Ideas?  Feedback? Would this be good?  What day are you free to do this?

How do I track all the test equipment

Posted August 29, 2007 by Brent
Categories: Test Harness

I have ideas how to track it but I want to track it in a way that others could use the tracking tools also. The key is setting it up in a way that the test harness can see what is available, working then reserve it for the duration of the test. At Convio we had test systems that got a build and were set up a certain way that was needed to run most of the tests. If we had automated tests that would have been the target, the parameters you feed in are: host name, db credentials and possibly what browser platform to use.

At Uplogix I have 2 pieces of equipment to test: and Envoy that you ssh into to control various devices and an EMS that has a web front end that controls many Envoys. The configuration is complicated by what devices are physically connected to the Envoys (routers, switches, GPS units, Unix machines or simulators for these devices) and how they get connected to the EMS.

I want to solve this in a generic way. Defining resource types (Envoy, EMS, Serial device, power controller,…) instances of those resources and how they are connected. Then I can write scripts to make sure the devices are functioning (they break a lot with what we do to them) configure them when new builds show up, and allow test cases to find what it wants then reserve it. That last part is the most interesting.

So I may have 7 Envoys of two different flavors, with 4 routers connected to them, 3 of them are connected to a power controller. A test case that needs to break into a router by power cycling the router and catching the prompt and sending a new config to it would have to look for this:

(Envoy: type=’TU301′, name=$envoy, reserved=false) and (2600: envoy=$envoy, power=$power, port=$port, reserved=false)

So prolog or an SQL inner join, it would match the name of the TU301 with the 2600’s envoy so it knew they are wired together (and not reserved by another test or person) and give you the name of the devices, the port id of the Envoy, and the power port id to pass to the test case.

In an SQL transaction it would have to reserve the unreserved equipment or roll back and wait or try a different set of gear. Once the tests are complete it would unreserve the 2 pieces of equipment.

When that TU301 test is done we now look for ‘Spitfire’ instead and run the test again with the new parameters. If we don’t find matching equipment we flag it as skipped due to lack of resources or busy resources for a person to fix.

For bonus points I can run the sanity tests on both devices after the test and if the router is in ROMMON mode or unresponsive I know what tests caused the problem and take the device out of service. Possibly disable the test too.

So now I have tests running every night

Posted August 28, 2007 by Brent
Categories: Test Harness

I have tests running every night, great. Now I need to monitor the results.

The tests are written in Perl and Expect.pm to test a CLI. The tests are not super important or contain any measurable coverage but allow me to see a problem right off the bat that needs to be solved now. I now have to go look at a file generated and see what the deltas are. Here is how I got to this point.

  • Create Perl libraries to figure out what build is on a machine and load a different one
  • Make an object that encapsulates all the Expect code to communicate with the device I am testing.
  • Generate tests using the object with Perl’s Test::More module.
  • Write a program that looks at Cruise Control build machine finds the latest build use the library to load this onto the test machine then run the tests.

So, here is the problem now, I am looking at a file every morning to see what the results are. I see this output:

ok 1 - timeout set to 120 minutes
ok 2 - timeout set to 5 minutes
ok 3 - timeout set to 101 minutes
ok 11 - timeout 5 minutes after disconnect
ok 12 - timeout set to 120 minutes

So I now want to move this to my database and have to program e-mail when there is a delta. I can look at the results with some CRUD scaffolding. Later, there will be web testing that will use Watir and be in Ruby.

How do I do this?

Write something like Test::More in Perl and Ruby that connects to a database?

Or extened Test::More to do this. What about Ruby? Extend RSpec to do this?

I like the power you get from tools like RSpec and Test::More but do they work well outside of a unit test environment?