Is the World of Mobile Testing Round? Ramblings of a Kitesurfer…

Whilst watching back some kitesurf vids the other day… it struck me that a birds eye view of the situation often reveals some startling results. Sometimes it is good to take stock… look around and re-confirm that the world is round. However… in the often complex world of mobile device testing, this seldom happens. It should do… and needs to!

Between you, me there are some high level ‘gotchas’ that only surface when you have spent quite a lot of time researching and looking at solutions. My check list is as follows:

Device or not to Device. That is the Question…

1. Need to target specific OS’s as they are all different.
   1. I.e. IOS, Android etc. Unless you have a lot of resource you will just not manage to test everything… for example blackberry and windows etc. etc.
2. Need to decide whether you are doing simulators/emulators or real devices.
   1. Or a mixture of both.
   2. A lot of people I have spoken to, forget or ignore the power of simulators/emulators for testing.Whilst Simulators/emulators are used in development, their usage is not always carried forward into the testing environment. No real reason for this I think, other than a lack of educational swap-over, from development to test… Simulators/emulators can appear to be complicated and technical to setup (hardware and software), but can add real benefit to your testing solution. Do not ignore.
3. If (you should be!) testing real devices… You need to target specific devices as again the differences are complex.
   1. E.g. Differences between Android devices will be present… Differences between Samsung Galaxy S2 and HTC equivalent, can be keyboard, navigation, screen resolution etc.
   2. Also remember that IOS is again different from everything… has no Adobe Flash Players etc.

Simulation versus Emulation… Not Assimilation!

1. Need to properly understand the differences between simulators and emulators.
   1. iPhone Simulator in Apples’s Xcode only mimics the software environment. NOT the hardware resources such as memory, disk space, processor speed etc.
   2. Android Emulator mimics the software and hardware environments found on actual devices. The drawback is that they can be more complex to setup as you have to target real device configurations etc. BUT they are more true life obviously…
      1. Note: Though android emulator emulates the ARM processors and certain hardware, it still doesn’t do a good job of matching the CPU performance.
2. Simulators/emulators require e.g SDK installation according to manufacturer OS you are testing.
   1. Android needs Android SDK with then optional device environments for Samsung / HTC etc.
   2. IOS simulator testing only works on Mac with Apple SDK (Free download) installed. Need to be on recent Apple OS to support latest Apple SDK and therefore get latest device simulators for IOS 5 / iPad 3 and iPhone 5 etc.
3. Be wary of free Javascript simulators, or web plugins etc. These are most likely not going to give you a true environment and are most likely to give false positives etc. when you are testing. Go the extra mile and get setup properly. There are very few shortcuts in this game…
4. Can be very difficult to load the application under test into the simulator or emulator correctly. Development help and planning often required. E.g. An apple developer license and Xcode environment is often required… or a windows development license and environment setup etc.

Conclusion… so far… Device testing is very important…

Hardware… go Hard or go Home!

1. Most device testing requires a USB connection… so you need to think about desktop PC’s with the ability to have drivers etc. installed on them.
   1. E.g. Advanced connection to Android phones require you to have a) Android SDK installed for ADB connection and b) Model drivers etc. E.g. Samsung Kies, or HTC Sync for USB device connection etc.
   2. E.g. For IOS, Apple requires iTunes.
2. Device testing will largely be played out on the PC desktop… so Citrix and Terminal services etc. will make life exceptionally difficult I would imagine.
3. Device testing is usually singular in operation. I.e. normally you will have a single virtual machine and then host, or single machine setup at any one time, for each device model you are testing. For Testing this means multiple licenses will need to be purchased to carry out multiple device testing.

Applications… Angry Birds or Angry Nerds!

1. Getting the application onto the device is problematic, whether you are dealing with real device testing or simulation/emulation.
2. Application load in development environment:
   1. Requires a close synergy with your development team to load application from development and test in a proper production like environment.
      1. All too easy to take short cuts on setting up a proper environment to make life easy for the development team. However mistakes here will give false positives and questionable test results.
   2. More suited to simulation/emulation than real device testing due to publishing lag on markets etc.
3. Application load from Market:
   1. Testing can invariably start quite late in the cycle as publishing to Google Play (Android Market) or iTunes can take a lot of time and patience.
   2. If you are testing production ready applications, and depending on the number of devices you are testing; be prepared for multiple sign ups to Google and iTunes etc.

Testing Tools… or Testing Fools… Do not always trust what it says on the tin!

1. IF you are still reading this… and can remember the silver bullet era of testing by HP… then read on Macduff! Because as Nasa would say… we have re-entry!
2. There are a lot of testing companies that are advertising mobile testing tools, but only a few companies that are actually building them.
   1. A few of the big boys e.g. HP… Borland… IBM are OEM white labelling other products. E.g. Borland’s Silk Mobile is actually Experitest SeeTest.
3. Cloud or no Cloud.
   1. Cloud – Positives are that you do not have to own the devices… Negatives are that you do not own the devices…
   2. I know that this is a flippant remark… but the feedback to me from customers who have gone down the cloud route is that it is far more costly in the long run, and prone to availability downtime and configuration nightmares for clean environment testing etc.
4. There are in my opinion 2 main divisions of testing tools…
   1. Object based application automation – Application under Test – This is where all the big boys are playing. With slick marketing and lots of images, this is advertised as a ‘quick’ (anything but, I would say!) method of getting up and running with your mobile test automation.
      1. Invasive Source Instrumentation (compile-time) code required. – In nearly all cases… you will need access to developers and development environments… This is so that you can get the specific test tool hooks and object recognition code embedded and compiled into your application under test… so that you can drive, test and receive results. This is an application under test scenario… not a system under test! As most of these tools require development of the testing into the application being tested, you need to really look at the tools you are evaluating and ascertain the level of technical implementation impact your testing will have on the application development.
      2. With so much embedded code compiled into your application to allow the automation to function… how on earth do you call this a clean testing environment?
      3. How do you test the production release? Do you leave the embedded testing code in place? Or do you strip your testing code out and then hope for the best when you load it into the market for release?
      4. Very difficult in most circumstances to test outside the application sandbox. I.e. Application interaction with other applications like email / SMS etc. is hard to achieve.
      5. Very difficult to test “off the market” applications developed by third party suppliers or OEM partners etc.
      6. Having the testing code in the application can allow however for greater ease when testing multiple devices using the same or similar test scripts etc.
      7. Often does not require a jail broken or rooted device.
      8. Object orientated approach is often very technical. Needs careful resourcing of your test team etc.
   2. Image based application automation – System under Test [T-Plan Robot Enterprise] – Tests the mobile device (System under Test) by connecting via WiFi or USB, to a VNC server, or other server, running on the device.
      1. Due to the presence of a VNC server, or other server to command the device (keyboard / mouse etc.), there are 2 main options:
         1. NON-Jailbroken:
            • For iOS you need an Apple Development License (very inexpensive & easy to enrol) to install (using Xcode & iTunes) the server components etc. (Not available in the App Store).
            • For Android you can choose between VNC Server, or use the  ADB (Android Debug Bridge with Android SDK, over USB or WiFi).
            • For other mobile environments such as Windows Mobile, Windows CE, Blackberry there are also VNC Server solutions available.
              • For Windows Phone life is tricky at the moment with a distinct lack of Microsoft & developer interest to provide tools for controlling applications, or the device, from a PC etc.
         2. Jailbroken (Root):
            • Rooting the device is a consideration that has to be planned for and decided upon. E.g. For iOS, whilst rooting and installing Veency on the device is difficult, the plethora of options this then gives in terms of fully controlling the device etc. is enormous. However rooting can invalidate your warranty so it would need to be sanctioned etc.
              For Android, as usual, the process is easy and the rewards are the same.
      2. Non-invasive – No application authoring to include test tool hooks etc. I.e. NO Source Instrumentation (compile-time) code required. Allows your ‘vanilla’ application to be received (from the developers or third party) / downloaded (from the market or test portal etc.) / installed (like it would be in a production environment) and tested. I.e. True production environment readiness can be achieved, as no foreign code has been included into the application for testing.
      3. Tests the application interaction with other application on the mobile device. As you are testing by connecting to the device… these tools truly test the system under test, and the application under test connectivity, to other applications on the device etc.
      4. Very easy to test outside the application sandbox. I.e. Application interaction with other applications like email / SMS etc. is hard to achieve.
      5. Very easy to test “off the market” applications developed by third party suppliers or OEM partners etc. Test per application, or test a suite of applications interaction synergy on the device.
      6. Multiple devices using the same script is supported with careful planning of the test image repositories etc.
      7. This GUI lead testing, tends to be much less technical than the development orientated approach of object recognition. Resourcing can be easier, as often particular skill sets are not required.

Futher Reading: Unbiased white paper on object versus image