Archive for the ‘General Test’ category

Hard to Ignore Build Status

November 4, 2016


When the number of builds and tests you are looking after start to grow the dashboards are easy to forget.  A solution is to have some NeoPixel lights sitting in your group area that clearly indicate what is building, what is working and what isn’t working.

The goal here was to build something obvious that costs around $20 that could be seen in a team area.  I didn’t want the tech to be intrusive so I just read messages out of Slack from TeamCity then later Jenkins.  Because this is reading messages from a Slack bot anybody can set one of these up as long as they have permission to the slack channel and it doesn’t matter how many people are consuming these messages.

I would like to use a different controller to untether this from my computer as it shuts down when I go to meetings now.  The messages I get from the build bots can be hard to read when you are building pull requests and various branches.  It might need HipChat support too.



rspec and instance variables

April 4, 2011

I was working on watir koans at Watirday 2011 and ran into a snag.  I had assumed I did something careless and needed to look at it when I could focus better.  Later that night I noticed that night I realized I just didn’t understand the scoping rules.  I created a new watir-webdriver object in the before(:all) block and used it in the examples for the example group.  The problem started when I didn’t want to replicate finding the form in later examples so I created a new @f class variable in an example to hold the form but it was nil in the next example.

I was lucky to have an experienced ruby tester, Marek J, there to explain what had happened.  I heard about class variables, classes within classes and copies of variable appearing in the examples.  If you change the variable in one example you will see it go back to the original value in the next example.  This is rspec helping isolate the examples.

Simple right?  No. there is an exception you need to know right away.  If your variable is a reference to something, in the heap, then modifications to that data will be referenced through the class variable.

Ok, so simple variables will be reset but references to complex variables, like a hash, will change keep state between examples?  Well, yes and no, the final example below shows that changing the instance variable to point to a different hash will be reset between examples just like the simple data types.  You kinda have to know about pointer or references to get it.  This happens in Perl too so I understood.  I googled a little and didn’t see it explained so I wrote this blog entry.

The root of what I’m describing is instance variables in classes and subclasses. This is relevant to rspec because:

  • the describe block is magic that makes a class
  • the it block is magic that makes sub class of the describe class
  • the before and after blocks are not in a sub class they are part of the describe class

Because the tester doesn’t see the module or class in their code they may not know what is happening under the covers.  However, they do need to know the behavior or they may learn to just use globals.

require 'rubygems'
require 'rspec'

describe "rspec for simple variables" do
  before(:all) do
    $global   = 'sg'
    @@class   = 'sc'
    @instance = 'si'

 it "should see all simple before variables" do
   $global.should   == 'sg'
   @@class.should   == 'sc'
   @instance.should == 'si'

 it "should change all the simple before variables" do
   $global   = 'new global value'
   @@class   = 'new class value'
   @instance = 'new instance value'
   #modifying a copy of the one defined in the before block

   $global.should   == 'new global value'
   @@class.should   == 'new class value'
   @instance.should == 'new instance value'

  it "should verify changed simple variables in this example" do
    $global.should   == 'new global value'
    @@class.should   == 'new class value'
    @instance.should == 'si'
   # Surprise, @instance.should is still 'si' from the begin block

describe "instance references can be changed" do
  before(:all) do
    @instance_hash = {:i_live => "heap memory"}

  it "should see the heap variable" do
    @instance_hash.should == {:i_live => "heap memory"}

  it "should modify the heap variable" do
    @instance_hash[:i_live] = "modified hash"

  it "should see the instance new value via the instance var" do
    @instance_hash[:i_live].should == "modified hash"
    # What just happened? The string was not reset between
    # examples. It points back the same memory but the contents
    # of the memory have changed.

  it "should set the class hash to a different hash" do
    @instance_hash = {:i_live => 'new hash'}
    # This hash will be lost forever when @instance_hash
    # goes out of scope when this example finishes.

  it "should still have the modified hash from the before block" do
    @instance_hash[:i_live].should == "modified hash"
    # The previous example changed the reference of hash so it
    # was restored to point to the one from the begin block.

Sorry about the bad formatting, I don’t know what WordPress knobs to turn to make this page wider without paying money.

Where did IBM blow all that smoke?

September 26, 2007

I am waaaaaaay to busy to be distracted today. There is a release going out on Friday and like usual there is a lot of work to do. So when I got a Dr. Dobb’s e-mail with something that looked somewhat test oriented that pointed me to IBM I went. This paper seemed like it took testing one step farther than record and playback testing as it would try to figure which ‘test case’ you were executing and offer to finish it out for you. The more I read the more I felt like I was reading sci-fi. Then I got to this on page 4 of 4: “Although these are currently unimplemented, we will give a brief overview of some of the possible approaches” I now know where the smoke was getting blown. Wow, good luck with that! I feel like someone from the testing community should say, hey we are working on these problems… and they are real hard to solve.

Should there be a Watir users group in your town?

September 20, 2007

I think it is time for Austin to get a Watir users group.  Bret Pettichord checked the interest of this about a year ago, the response was good.  It may be even better to start one now.

In the last year I have seen some great advances of what to do with Watir to help people really take this technology and start writing practical tests very quickly.  Last week Hugh McGowan and Rafael Torres showed off a flexible data driven test system that made each row or column of a worksheet into a set of Rspec tests.  This was open source software on ruby forge.  Then shortly after that Mark Anderson used this to write a data drive test to hit google given search terms to verify the expected top url.  This code should be on the same ruby forge site now.  The demo was quick and impressive.
With this advances I saw someone ask a question on the Austin Watir mailing list how to install Watir on their system.  There was a quick answer.  The Wati-general mailing list is very busy these day with advanced and newb questions.

Now seems to be a good time to get an organized Watir group set up in Austin.  I would also like the idea of exporting our presentations (an importing as well) to user groups in other cities to get things moving that way as well.

I’m thinking a meeting would be called when we have a venue and 2 presenters and might go like this:

5:30-6:15 Early arrival, people can bring questions to see if a more experienced user can help them.  Possibly as advanced as installing Ruby/Watir/Test tools on the machine or a recent user-group presentation demo.

6:15-6:30 Regular arrival, people can announce that they are looking for Watir users and testers or are looking for work in the field.

6:30-7:00 Watir report – Someone reports experience with the Watir or related tools at work.

  •   This was the problem.
  •   We did this
  •   This worked/this didn’t

7 :10-7:50 Demo – Showing off a new tool, module or library

  • See Watir in action
  • Learn about coding techniques or a tools
  • Later you can down-load and try out something

Ideas?  Feedback? Would this be good?  What day are you free to do this?

First blog

July 11, 2007

Last night I was talking with other testers over a few beers and realized I was a non-blogging dinosaur that still read magazines. I was also convinced that I was not a testing dinosaur and that I had something to contribute. So here I am, I updated my RSS feeds to look at things other than photography and cycling and I will start writing. Really, I promise.

My areas of test automation are [I think I need better names]:

  1. Full automation – instead of manual testing helpers. This includes hooks into cruise control to start tests, cron, test lab setup, tear down and informing people of unexpected results. I have nothing against test that help manual testers it just isn’t interesting to me now. At Uplogix I have a lab management issue tracking and all the equipment that I test with.
  2. Test harnesses – This is the part of the test system that someone else can use regardless of what they are testing. Your test cases follow patterns indicating passing/failing, expected/unexpected results. When a test case gives up due to environment or failures and when it continues. It may have to activate hardware or virtual machines to connect to a server and get tests to run on that platform. Unit testers have frameworks like RSpec for Ruby, JUnit for Java or Test::More for Perl but I don’t know of an open source harness or framework for system testing. Something that would understand that the same test needs running on Windows ME/IE5 and Windows Vista Business/IE6 are different tests.
  3. Test infrastructure – how you organize your test code so the proper code is in the right place.
    • Abstraction layer giving you an API, in a ‘real language’) to write your test cases to. This might deal with the UI, WebServices, Rest an underlying RDB or a file system.
    • Test code the steps – The bulk of your testing intellectual property safe from UI changes or the technology that connects to it.
    • Test Harness – described above
  4. Automated test organization – Version tracking tests and the abstraction layer, tracking known failures, and test results.
  5. Virtualization – using technology, like VMWare, to test different OS/Browser combinations in a clean room environment. Virtualizing machines for upgrade could pay off well too.
  6. Web testing, Watir in particular. Working with the ruby test-system group, to test the reference web application.
  7. Command line testing – I am currently using Perl’s module to write an abstraction layer.