Hard to Ignore Build Status

Posted November 4, 2016 by Brent
Categories: General Test



When the number of builds and tests you are looking after start to grow the dashboards are easy to forget.  A solution is to have some NeoPixel lights sitting in your group area that clearly indicate what is building, what is working and what isn’t working.

The goal here was to build something obvious that costs around $20 that could be seen in a team area.  I didn’t want the tech to be intrusive so I just read messages out of Slack from TeamCity then later Jenkins.  Because this is reading messages from a Slack bot anybody can set one of these up as long as they have permission to the slack channel and it doesn’t matter how many people are consuming these messages.

I would like to use a different controller to untether this from my computer as it shuts down when I go to meetings now.  The messages I get from the build bots can be hard to read when you are building pull requests and various branches.  It might need HipChat support too.




Is it time to test with AWS Lambdas?

Posted September 7, 2016 by Brent
Categories: Uncategorized

If you application is written to run in Amazon AWS are Lambdas not the perfect way to run your system and integration tests?  Currently you are limited to Node, Python or Java for writing Lambdas but you have 5 minutes to execute your test and you can run 100 of them concurrently.  Unlike an an EC2 instance the minimum charge is 100 milliseconds plus $0.20 for every million calls after the first million.  The prices are here:https://aws.amazon.com/lambda/pricing/  If you wanted to save money you would have the tests cascade from one lambda to another and short circuit tests when other fail.

Testing API calls will only take a few milliseconds but even UI tests might run under the 300 second limit but I have worked at places where that be hard to do but it would be headless.

If I only had a AWS app to test.

I was once a mobile tester

Posted December 22, 2015 by Brent
Categories: Uncategorized

For two years I tested Android and iOS applications with Appium then Calabash.  I wrote about the Calabash here: mobileautotest.  While this is no longer my day job I still do find it interesting.  I wanted to post this here in the event anyone out there is dealing with the tough world of mobile test automation.

While there is some value in test automation on this platform it is a tough place to get a good ROI.  Appium was good until the day it wasn’t and Calabash was better but knowing what I know now I’d probably pick a different platform.

Don’t put spaces in your Jenkins job names

Posted September 24, 2015 by Brent
Categories: Uncategorized

Use a dash or underscore instead of a space in your Jenkins job names.  When you put a space in them you get spaces in the workspace.  While it is annoying to deal with this directory name on the command line you can get by.  Most of your code will be fine, but some shell scripts may not work.  We recently moved from Crashlytics to Fabric pod and some file script is derived data. The space in $PODS_ROOT broke this:

${PODS_ROOT}/Fabric/Fabric.framework/run some big strings

Another benefit is that URLs for you jobs will not have %20 sprinkled in them.

I moved my config into yaml files

Posted January 23, 2014 by Brent
Categories: Uncategorized

My rspec tests used to have a before each that looked like this:

 @driver = HAmvc::Configuration.instance.get_driver :ios
 @welcome = MobileTest::Welcome_controller.new @driver
 @signin = MobileTest::SignInController.new @driver
 @menu = MobileTest::MenuController.new @driver

The controllers would be constructed like this

def initialize browser
 @view = SignIn_view.new browser

Now the before all looks like this:

 @welcome = MobileTest::Welcome_controller.new
 @signin = MobileTest::SignInController.new
 @menu = MobileTest::MenuController.new

And the the controller does this

 def initialize view = SignIn_view.new
   @view = view

The view class now knows how to get a driver from the singleton that has them. So the an environment variable set to ios_stage would load up the center chunk of this:

 :env: stage
 :os: android
 :artifact: http://jenkins-machine/job/android-stage-build/lastSuccessfulBuild/artifact/HomeAway-app/build/apk/HomeAway-app-release.apk
 :locale: en_US
 :env: stage
 :os: ios
 :artifact: http://jenkins-machine/job/HomeAway-iOS/DEPLOY_ENV=Stage/lastSuccessfulBuild/artifact/artifacts/homeaway-internal-STAGE.ipa
 :locale: en_US
 :env: stage
 :os: ios
 :artifact: /Users/blavelle/iOS.app
 :locale: en_US

I’m much happier with this setup but now the question is, how do I iterate over configurations?  Should I do it within a ruby or have a different Jenkins job for each configuration in that yaml file?

Continuous Android testing

Posted December 30, 2013 by Brent
Categories: Uncategorized

Getting my tests to run when Jenkins builds the Android application has been a challenge.  I had this running with Genymotion 1, on a Mac Mini, from a Jenkins agent then the upgrades came.  First Genymotion went to version 2 and the stability dropped.  I had to stop and start things more often.  So I ditched Genymotion and went to the emulator that comes with the Android SDK.  Finally we got new iMacs and they are running Mavericks instead of Mountain Lion.  Nothing worked for a while, time to rebuild.

I built a few images for the emulator and got the emulator working with my application.  Running this bestest macbook pro retina the tests time out.  I used these settings from the appium-discuss group: https://github.com/appium/ruby_console/blob/master/img/avd_settings.png but it was too slow to run the tests that had been passing with genymotion.  Here is that thread where I posted timings: https://groups.google.com/forum/#!searchin/appium-discuss/genymotion/appium-discuss/p0VbkiilPIw/rz2Ry8KppIQJ At this point I’ve decided that the Google emulator is not good.  It seems like the network is way too slow and the UI is just a little slow but I don’t want to waste any more time on this.  So I am left with the real device or going back to genymotion.  Genymotion is €99-299 if I go with that option or $600 or so for a Nexus device and the hassle of having to secure the device at my desk.

Genymotion again:

I rebooted my machine, grabbed genymotion 2.0.3 and ran one of the 1.X images that I had sitting around from the good old days.  I got an error but the emulator fired up very quickly and I was able to run my tests without them failing.  I will need to keep an eye on this to make sure that this solution is stable again.  Our application needs the google apps to fully function so the new images are not what I want to run with.

A common problem that will arise is that the tests will not see a physical device or an emulated one from time to time.  You can verify that this is a problem by running this command and seeing no attached devices:

adb devices

If there are devices these two command usually fix the issue:

adb kill-server
adb start-server

The final step was getting the latest 32-bit (Chrome, why do you hate OSX?) on my laptop and running the jenikns jnlp and binding my job to it.

Next, figure out what is wrong with having the agent run in Mavericks.

Continuous integration and testing

Posted December 26, 2013 by Brent
Categories: Uncategorized

Tags: , , , , ,

Automated tests that are not running in your continuous integration system don’t count.

It is difficult to keep my automated tests running in Jenkins.  The iOS tests use the Apple emulator that the developers use.  I, as well as many of the Android testers used Genymotion until 2.0 came out.  The tests worked when the build machine was running Mountain Lion but not with Mavericks.  Another issue is getting the build bits from the the Jenkins projects that make the applications to the projects that test the applications.  For the HomeAway application there is a single test job for both application projects.

Even though the Android and iOS applications are tested with the same project and the same code testing them with Jenkins are different problems with less in common than I would expected.  Initially I wanted to cover both of these topics with one blog post but I will probably post about these separately and several times each.