Mobile Testing Guide
The Beginner’s Guide for Mobile Application Testing
This guide is an introduction to mobile application testing. Learn what mobile testing is, why it is important, the challenges that are unique to mobile testing, and how to test native mobile applications as well as mobile and hybrid web applications. If you are brand-new to software testing, it is recommended that you start with the Ranorex Beginner’s Guide for User Interface Testing before reading this guide.
What is a mobile application?
Types of mobile applications
Native apps are those written for a specific platform, such as apps written in Objective-C or Swift for iOS devices or in Java for Android devices. Native apps are distributed to users through an application store, generally provide the best performance for a mobile device, and have access to all of the features of the device.
Hybrid apps have a native front-end and a cloud-based backend. The native front-end provides access to device features, while the backend can make use of web resources such as online databases. For example, an online store may provide a hybrid app that allows users to search the online store and make purchases from their mobile devices, or a photo-sharing app may allow users to take photos, edit them, and then store them in the cloud for sharing. Like fully native apps, hybrid apps are distributed to users through an application store. Tools such as Amazon Web Services (AWS), Mobile Hub, or Google Firebase support development of both hybrid and mobile web apps.
Why do mobile application testing?
An app that doesn’t meet Apple’s standards for quality won’t even make it past the review process and into the application store. Mobile app users are quick to delete applications that freeze, crash, are slow to load, or have a poor user experience – and leave bad reviews in the application store. With tens of thousands of apps available to users, even a few bad reviews can be the death knell for an application. Once a defect in a native or hybrid app has made its way onto a mobile device, fixing it is a time-consuming effort that includes re-submitting the app to the application store, waiting for review by Apple and waiting for the user to download the update.
Effective testing is critical to minimize defects and improve quality in mobile applications, leading to apps that keep users engaged and earn high ratings from users. For this reason, mobile application tests must be well-designed to cover not just the functionality of the app, but also its ease of use and performance. Users won’t tolerate apps that use unreasonable amounts of data or that drain their device’s battery. The app must also be able to respond gracefully to events such as turning the device off, putting it into airplane mode, or connecting to another device via WiFi, USB or Bluetooth.
Looking deeper: O/S variety
Smartphone Market Share
Android is the leader with about 85% of the smartphone market, while iOS has 15%.
Mobile Market Share
When tablets are included, Android has about 65% of the total mobile market, with iOS at 33%.
Android Versions in Use
Worldwide, there are more than two billion active Android devices in use, but significant percentages are not on the latest version. This is because while Apple controls what version of its OS is sold with new devices, Google does not, but instead relies on device vendors to update their new phones and tablets. The result is a fragmented install base, which creates a significant challenge for application testing. As of August 2017, 15% of Android devices are running version 4.4 KitKat, 29% are running 5.0 or 5.1 Lollipop, 32% are running 6.0 Marshmallow, and 16% are running 7.0 or 7.1 Nougat, with the remainder running versions prior to 4.4 KitKat.
Mobile application testing levels: when to test
Test your mobile app’s design to ensure that the application has the right functionality and ease-of-use to keep users engaged. For example, the content should be responsive to the size and orientation of the device so that users do not have to scroll, and the app should respond appropriately to touch controls. Apple offers detailed guidelines for UI design including specific suggestions for button and text size, use of progress indicators and alerts, application settings, and controls. Likewise, Google offers a wealth of design resources for Android developers. These resources are extremely beneficial for QA personnel responsible for reviewing the design and usability of mobile applications.
A common approach to improving the quality of application design is A/B testing, which compares two designs to see which provides the better user experience (UX). In A/B testing, two groups of users each evaluate a feature of an app, such as its menus or registration process. Testers observe the users
Mobile application testing types: what to test
Performance testing is critical in mobile environments, as users will be quick to delete applications that run slowly, consume too much battery or data, or consume memory which can lead to unexpected shutdowns. The list below provides a checklist of items for performance testing:
- Application speed, including how quickly the application opens, how quickly it retrieves or stores information from back-end services, and whether there are any unexpected lags in functionality
- Battery usage, including not only how much battery the mobile application demands but also how the application performs in a low-battery situation
- Data consumption demands during typical use scenarios, as well as in edge cases of minimal and maximum use
- Low-level resource/memory usage/memory leakage. Memory leaks occur when an application feature fails to release memory when it is no longer needed, which can lead to application crashes.
Regression testing involves re-running tests that were successful in previous versions of the mobile application. This is done to ensure that new code hasn’t re-introduced old defects, called “regressions.” Because regression tests are performed over and over again, they are ideal candidates for automation. To learn more about regression testing, refer to our Regression Testing Guide.
Mobile testing techniques: how to test
Mobile application testing can be either manual or automated, and performed on simulators, emulators or physical devices. Manual testing on physical devices provides the most realistic recreation of a user’s environment but can be time-consuming and expensive. Automated mobile testing can improve testing efficiency to minimize the time to release. Once a test has been automated, it can be performed again with each new version of an application, to ensure that new code does not introduce regressions.
Simulators reproduce the software environment of an app and are suitable for preliminary testing and sanity testing. Apple provides a simulator as part of its Xcode developer toolkit. One drawback of simulators is that they don’t always behave like real devices, especially in the area of low-level resources like memory and battery. Emulators go one step further and attempt to reproduce the entire environment of the device including hardware. The Android Studio includes an Android emulator that “provides almost all the capabilities of a real Android device. You can simulate incoming phone calls and text messages, specify the location of the device, simulate different network speeds, simulate rotation and other hardware sensors, access the Google Play Store, and much more.”
If possible, test your application first on simulators or emulators, to eliminate as many defects as possible prior to going through the installation process on physical devices. Begin testing with smoke and sanity tests to verify that basic functionality is in place before committing valuable time and resources to further testing.
Physical devices provide the most realistic environment for testing your application. It is especially important to conduct performance testing on physical devices to ensure that users will be satisfied with your application’s speed, ease-of-use, and consumption of low-level resources.
Provisioning of Apple devices
Native or hybrid apps for Apple devices must be provisioned before they can be tested on physical devices. Provisioning is the process of preparing and configuring an app to launch on devices and use app services, which includes installation of the app, configuration of user accounts, and code signing of the app. Provisioning is accomplished in the Apple Xcode development environment. No provisioning is necessary for mobile web applications, as nothing is installed on the target device.
On Apple iOS and tvOS devices, provisioning begins with registering test devices in an Apple developer account. The next step is creating an iOS application archive file (.ipa) containing the app and creating an ad hoc provisioning profile, which supports the installation of the beta version of the app on designated devices. The profile includes the ID of the application, the unique device ID (UDID) of the test devices, and a distribution certificate. Once these exist, the app can be installed by connecting the iOS or tvOS devices directly to a Mac, or using iTunes for iOS and watchOS devices. Apple Configurator can also be used to install an app on all three platforms.
Preparation of Android devices
Considerations in selecting mobile testing tools
Automating mobile tests is essential to provide the fast feedback needed in continuous testing environments. There are both open source and proprietary solutions available. Appium is a leading open source tool for testing native, mobile web and hybrid applications on iOS, Android and Windows. Appium uses the popular, open-source Selenium WebDriver protocol.
When selecting a tool there are several aspects to consider:
- Can the automation tool execute tests in parallel on multiple devices?
- Does it support both simulators/emulators and real devices?
- Does it offer reusable code modules and provide data-driven testing?
- Does it offer support for native apps, mobile web apps, and hybrid apps for all major platforms?
- Does the tool have a design once, run everywhere approach with unique configurations for each endpoint?
- Does the tool support leading open source solutions including Selenium Webdriver nodes/grids and Appium?
- Does the tool support JSON capabilities for instrumenting hybrid and mobile apps?
- Does the tool offer GUI object recognition or is it limited to image recognition?
Tip: If you are ready to start automating your mobile application testing, download the free trial of Ranorex Studio today. Or, contact one of our test automation experts.