Archive for December 2013

Continuous Android testing

December 30, 2013

Getting my tests to run when Jenkins builds the Android application has been a challenge.  I had this running with Genymotion 1, on a Mac Mini, from a Jenkins agent then the upgrades came.  First Genymotion went to version 2 and the stability dropped.  I had to stop and start things more often.  So I ditched Genymotion and went to the emulator that comes with the Android SDK.  Finally we got new iMacs and they are running Mavericks instead of Mountain Lion.  Nothing worked for a while, time to rebuild.

I built a few images for the emulator and got the emulator working with my application.  Running this bestest macbook pro retina the tests time out.  I used these settings from the appium-discuss group: https://github.com/appium/ruby_console/blob/master/img/avd_settings.png but it was too slow to run the tests that had been passing with genymotion.  Here is that thread where I posted timings: https://groups.google.com/forum/#!searchin/appium-discuss/genymotion/appium-discuss/p0VbkiilPIw/rz2Ry8KppIQJ At this point I’ve decided that the Google emulator is not good.  It seems like the network is way too slow and the UI is just a little slow but I don’t want to waste any more time on this.  So I am left with the real device or going back to genymotion.  Genymotion is €99-299 if I go with that option or $600 or so for a Nexus device and the hassle of having to secure the device at my desk.

Genymotion again:

I rebooted my machine, grabbed genymotion 2.0.3 and ran one of the 1.X images that I had sitting around from the good old days.  I got an error but the emulator fired up very quickly and I was able to run my tests without them failing.  I will need to keep an eye on this to make sure that this solution is stable again.  Our application needs the google apps to fully function so the new images are not what I want to run with.

A common problem that will arise is that the tests will not see a physical device or an emulated one from time to time.  You can verify that this is a problem by running this command and seeing no attached devices:

adb devices

If there are devices these two command usually fix the issue:

adb kill-server
adb start-server

The final step was getting the latest 32-bit (Chrome, why do you hate OSX?) on my laptop and running the jenikns jnlp and binding my job to it.

Next, figure out what is wrong with having the agent run in Mavericks.

Advertisements

Continuous integration and testing

December 26, 2013

Automated tests that are not running in your continuous integration system don’t count.

It is difficult to keep my automated tests running in Jenkins.  The iOS tests use the Apple emulator that the developers use.  I, as well as many of the Android testers used Genymotion until 2.0 came out.  The tests worked when the build machine was running Mountain Lion but not with Mavericks.  Another issue is getting the build bits from the the Jenkins projects that make the applications to the projects that test the applications.  For the HomeAway application there is a single test job for both application projects.

Even though the Android and iOS applications are tested with the same project and the same code testing them with Jenkins are different problems with less in common than I would expected.  Initially I wanted to cover both of these topics with one blog post but I will probably post about these separately and several times each.

I’m now a mobile tester

December 24, 2013

I’ve decided to take a job writing automated tests for HomeAway’s mobile applications.  So I’ve gone from using Watir and Watirmark to something new.  What I have to test  now can be broken down to these parts

  1. An iOS native application
  2. An Android native application
  3. A set of APIs that the native application uses for data CRUD
  4. An app for owners that is written using Phonegap

I started about 6 months ago and have been learning the business and the applications as well as looking at ways of testing that takes what I have seen work in the past.  When I first started writing proof of concept tests I used Appium Selenium and Ruby for testing 1 and 2 above.  The first big discovery is that I can’t use IDs to get at the elements.  I have to use the accessibilityIdentifier on iOS and the contentDescription on Android.  While the applications are developed by two different sets of developers and are not the same I was able to do some useful testing with the exact same test code.  To do this I had to get both dev environments up and running on my machine and put in the same accessibility strings in the two systems.  I imagine there will be places where UI code will look like

if platform == 'iOS'
do this
else
do that
end

I have not had to do that yet.

For testing the APIs I am using net/https and json.  This testing was easy to get going and is in TeamCity running after the API it is testing builds.

The Phonegap application is interesting.  I spent a couple of days trying to use Appium or variations of Appium to get at the application bit it reminded me of testing flash applications embedded in HTML  So, I am running the application as a web application on the local machine and using Selenium to drive a web browser and that is working now.

All of this code uses a Watermark-like MVC approach to test their parts with the exception of the API tests, in that case the View code is just the JSON and the controller just calls out to JSON instead of a normal view.  The JSON controller is subclassed from the normal controller and probably needs a little work.