Archive for the ‘Uncategorized’ category

Is it time to test with AWS Lambdas?

September 7, 2016

If you application is written to run in Amazon AWS are Lambdas not the perfect way to run your system and integration tests?  Currently you are limited to Node, Python or Java for writing Lambdas but you have 5 minutes to execute your test and you can run 100 of them concurrently.  Unlike an an EC2 instance the minimum charge is 100 milliseconds plus $0.20 for every million calls after the first million.  The prices are here:https://aws.amazon.com/lambda/pricing/  If you wanted to save money you would have the tests cascade from one lambda to another and short circuit tests when other fail.

Testing API calls will only take a few milliseconds but even UI tests might run under the 300 second limit but I have worked at places where that be hard to do but it would be headless.

If I only had a AWS app to test.

Advertisements

I was once a mobile tester

December 22, 2015

For two years I tested Android and iOS applications with Appium then Calabash.  I wrote about the Calabash here: mobileautotest.  While this is no longer my day job I still do find it interesting.  I wanted to post this here in the event anyone out there is dealing with the tough world of mobile test automation.

While there is some value in test automation on this platform it is a tough place to get a good ROI.  Appium was good until the day it wasn’t and Calabash was better but knowing what I know now I’d probably pick a different platform.

Don’t put spaces in your Jenkins job names

September 24, 2015

Use a dash or underscore instead of a space in your Jenkins job names.  When you put a space in them you get spaces in the workspace.  While it is annoying to deal with this directory name on the command line you can get by.  Most of your code will be fine, but some shell scripts may not work.  We recently moved from Crashlytics to Fabric pod and some file script is derived data. The space in $PODS_ROOT broke this:

#!/bin/sh
${PODS_ROOT}/Fabric/Fabric.framework/run some big strings

Another benefit is that URLs for you jobs will not have %20 sprinkled in them.

I moved my config into yaml files

January 23, 2014

My rspec tests used to have a before each that looked like this:

 @driver = HAmvc::Configuration.instance.get_driver :ios
 @welcome = MobileTest::Welcome_controller.new @driver
 @signin = MobileTest::SignInController.new @driver
 @menu = MobileTest::MenuController.new @driver

The controllers would be constructed like this

def initialize browser
 @view = SignIn_view.new browser
end

Now the before all looks like this:

 @welcome = MobileTest::Welcome_controller.new
 @signin = MobileTest::SignInController.new
 @menu = MobileTest::MenuController.new

And the the controller does this

 def initialize view = SignIn_view.new
   @view = view
 end

The view class now knows how to get a driver from the singleton that has them. So the an environment variable set to ios_stage would load up the center chunk of this:

android_stage:
 :env: stage
 :os: android
 :artifact: http://jenkins-machine/job/android-stage-build/lastSuccessfulBuild/artifact/HomeAway-app/build/apk/HomeAway-app-release.apk
 :locale: en_US
 :device:
ios_stage:
 :env: stage
 :os: ios
 :artifact: http://jenkins-machine/job/HomeAway-iOS/DEPLOY_ENV=Stage/lastSuccessfulBuild/artifact/artifacts/homeaway-internal-STAGE.ipa
 :locale: en_US
brent_ios_local:
 :env: stage
 :os: ios
 :artifact: /Users/blavelle/iOS.app
 :locale: en_US

I’m much happier with this setup but now the question is, how do I iterate over configurations?  Should I do it within a ruby or have a different Jenkins job for each configuration in that yaml file?

Continuous Android testing

December 30, 2013

Getting my tests to run when Jenkins builds the Android application has been a challenge.  I had this running with Genymotion 1, on a Mac Mini, from a Jenkins agent then the upgrades came.  First Genymotion went to version 2 and the stability dropped.  I had to stop and start things more often.  So I ditched Genymotion and went to the emulator that comes with the Android SDK.  Finally we got new iMacs and they are running Mavericks instead of Mountain Lion.  Nothing worked for a while, time to rebuild.

I built a few images for the emulator and got the emulator working with my application.  Running this bestest macbook pro retina the tests time out.  I used these settings from the appium-discuss group: https://github.com/appium/ruby_console/blob/master/img/avd_settings.png but it was too slow to run the tests that had been passing with genymotion.  Here is that thread where I posted timings: https://groups.google.com/forum/#!searchin/appium-discuss/genymotion/appium-discuss/p0VbkiilPIw/rz2Ry8KppIQJ At this point I’ve decided that the Google emulator is not good.  It seems like the network is way too slow and the UI is just a little slow but I don’t want to waste any more time on this.  So I am left with the real device or going back to genymotion.  Genymotion is €99-299 if I go with that option or $600 or so for a Nexus device and the hassle of having to secure the device at my desk.

Genymotion again:

I rebooted my machine, grabbed genymotion 2.0.3 and ran one of the 1.X images that I had sitting around from the good old days.  I got an error but the emulator fired up very quickly and I was able to run my tests without them failing.  I will need to keep an eye on this to make sure that this solution is stable again.  Our application needs the google apps to fully function so the new images are not what I want to run with.

A common problem that will arise is that the tests will not see a physical device or an emulated one from time to time.  You can verify that this is a problem by running this command and seeing no attached devices:

adb devices

If there are devices these two command usually fix the issue:

adb kill-server
adb start-server

The final step was getting the latest 32-bit (Chrome, why do you hate OSX?) on my laptop and running the jenikns jnlp and binding my job to it.

Next, figure out what is wrong with having the agent run in Mavericks.

Continuous integration and testing

December 26, 2013

Automated tests that are not running in your continuous integration system don’t count.

It is difficult to keep my automated tests running in Jenkins.  The iOS tests use the Apple emulator that the developers use.  I, as well as many of the Android testers used Genymotion until 2.0 came out.  The tests worked when the build machine was running Mountain Lion but not with Mavericks.  Another issue is getting the build bits from the the Jenkins projects that make the applications to the projects that test the applications.  For the HomeAway application there is a single test job for both application projects.

Even though the Android and iOS applications are tested with the same project and the same code testing them with Jenkins are different problems with less in common than I would expected.  Initially I wanted to cover both of these topics with one blog post but I will probably post about these separately and several times each.

I’m now a mobile tester

December 24, 2013

I’ve decided to take a job writing automated tests for HomeAway’s mobile applications.  So I’ve gone from using Watir and Watirmark to something new.  What I have to test  now can be broken down to these parts

  1. An iOS native application
  2. An Android native application
  3. A set of APIs that the native application uses for data CRUD
  4. An app for owners that is written using Phonegap

I started about 6 months ago and have been learning the business and the applications as well as looking at ways of testing that takes what I have seen work in the past.  When I first started writing proof of concept tests I used Appium Selenium and Ruby for testing 1 and 2 above.  The first big discovery is that I can’t use IDs to get at the elements.  I have to use the accessibilityIdentifier on iOS and the contentDescription on Android.  While the applications are developed by two different sets of developers and are not the same I was able to do some useful testing with the exact same test code.  To do this I had to get both dev environments up and running on my machine and put in the same accessibility strings in the two systems.  I imagine there will be places where UI code will look like

if platform == 'iOS'
do this
else
do that
end

I have not had to do that yet.

For testing the APIs I am using net/https and json.  This testing was easy to get going and is in TeamCity running after the API it is testing builds.

The Phonegap application is interesting.  I spent a couple of days trying to use Appium or variations of Appium to get at the application bit it reminded me of testing flash applications embedded in HTML  So, I am running the application as a web application on the local machine and using Selenium to drive a web browser and that is working now.

All of this code uses a Watermark-like MVC approach to test their parts with the exception of the API tests, in that case the View code is just the JSON and the controller just calls out to JSON instead of a normal view.  The JSON controller is subclassed from the normal controller and probably needs a little work.