Are you aware of the trade-off in your mobile test strategy

Developer make use of emulators to test mobile applications in addition to their own handsets.  But QA process wants to get closer  to the actual platform and purchase devices might not make sense for enterprise due to practical reasons like  cost and availability.

J2ME Emulators used to be PC based application to simulate the handset by mapping each attribute and the respective behavior of the real handset.  Newer emulators tend to take a different approach and actually compile the handset to work under a PC operating system, creating a  virtual test execution environment. They still lack features that exist only in mobile devices, cellular networks, accelerometer, quad core CPU, separate GPU and so on.

I see the advantage of the emulator is developer productivity that comes through debugging of your application on the emulator . They also help to simulate hard-to reproduce scenarios like low battery or GPS coordinates that a real device finds it tricky to support. The testers need to decide whether to perform test on a real device or reject the app to development seeing failures on emulator. Here are reason to test on real device beyond emulator

  • False positive situation: By performing only emulator based testing, one can reach  a “false positive” situation. Business hates this scenario for failure in reality and  in terms of cost. Let us take Android and there are many android devices and they have many variables. if we acknowledge that Software contains bugs, the issue or error in one particular android device might no appear on a different model. What is the business risk  in this scenario? Does it  impact on customer loyalty? If you are a bank or insurance company or enterprise rolling out mobile apps and services, you need to answer these questions.
  • Difference in Networking of device: Mobile emulators run on PC and and connect to the LAN to access internet and via corporate firewall. But real devices connect to the radio interface and from there access internet.  Effectively low level networking stack is different and can impact the behavior of the application in another way. The mobile OS has complete stack. Which one should you use?  On a different note, some mobile protocols might be prevented by local IT, while testing on emulator in PC.
  • Network -related events: Event like incoming call and incoming message must be test to determine the impact on the application and cannot be done in emulator. The impact of  different network technologies must be tested for network quality like carriers, states, countries, regions and even highly populated areas.
  • Device Constraints:  Mobile devices come with limited CPU power and memory (extend for graphic chipset, input devices, etc.). An application running on a powerful quadcore intel based CPU with 4 GB memory does not reflect the use experience on the handset with limited capability. There can also be scenarios where emulator will demonstrate a poor performance.

Let us assume the enterprise buys real devices. In a lot of enterprises and departments, USB ports are locked or disabled.  This brings up security related issues how to test connecting  to PC. In addition, what happens if the mobile device is stolen and unauthorized person gets access.

I recommend a hybrid approach.  Please discuss the trade-off with management on the cost of mobile testing and risk of releasing a faulty mobile application to the market.

  • Perform actual developer testing on emulator( with addition of one real device)
  • Perform sanity and regression testing on real handsets.
  • For automation testing, please try a minimum of 3 real devices. Do not trust the automated results on emulator to be always accurate. Make sure that  the automation scripts can be used both with  emulator and real devices with minimal or no change to scripts.
  • Identify early the  critical flows of the app and dangerous zones for the app between emulator and the real device.

Some pointer for the  mobile application life management(ALM)

  1. Collaboration is “must to have” and not only “very important”
  2. What to do when you find issues on mobile platform is not simple like “Screen Print” and attach a screen shot. Train testers to collect logs and files or use another mobile to take picture.
  3. Be ready to triage a large number of  “not reproducible” or “not sufficient information” cases in the defect tracking system
  4. If your target market is U.S, how to test from India. You can make use of cloud service based access to real devices for testing.
  5. Do not  under-estimate the logistics involved in mobile testing. For example, batteries drain out when you want to test and go home.  You also realize that SIM card has no money. You also learn that you have switched off wi-fi and the GPRS usage increased and your bill has shot up.
  6. Do not trust the testing done by developer and always make sure that secondary person tests
  7. Ask product owner for a set of reference devices for the development team to test before release time. They can be “Mandatory” devices, “Largely available” Devices. Selection of these devices need to be updated on daily basis.
  8. The cost and time to fix an issue after  publishing to app store increases significantly. You need to decide whether the error is major that the app needs to be resubmitted to app store for re-certification.
  9. Be ready to adapt for rapid changes. What is found today might not exist tomorrow.
  10. Business owners and  Product owners are not 100% equipped to describe mobile application features and seeing features in action enables better analysis.Hence shorter cycles can help to avoid large gaps between versions and market requirements and also feedback  from the product owner.