Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
Mobile Testing Guide
The Beginner’s Guide for Mobile Application Testing
This guide is an introduction to mobile application testing. Learn what mobile testing is, why it is important, the challenges that are unique to mobile testing, and how to test native mobile applications as well as mobile and hybrid web applications. If you are brand-new to software testing, it is recommended that you start with the Ranorex Beginner’s Guide for User Interface Testing before reading this guide.
What is a mobile application?
“Mobile application” is a broad term that includes three main categories of software: native applications, mobile web applications, and hybrid applications.
Types of mobile applications
Native apps are those written for a specific platform, such as apps written in Objective-C or Swift for iOS devices or in Java for Android devices. Native apps are distributed to users through an application store, generally provide the best performance for a mobile device, and have access to all of the features of the device.
As an alternative to developing software for specific devices, many developers prefer the write-once-run-anywhere approach of mobile web apps that run in a web browser. Leading mobile web application development languages include HTML5, Javascript, and Adobe Flash/Flex. Users can access web apps without having to first download them in an application store. Mobile web apps must be compatible with all of the major web browsers supported on mobile devices, including Chrome, Safari, Firefox, and Opera. A key drawback of mobile web apps is that they have a limited ability to access device features such as GPS, camera, and microphone.
Hybrid apps have a native front-end and a cloud-based backend. The native front-end provides access to device features, while the backend can make use of web resources such as online databases. For example, an online store may provide a hybrid app that allows users to search the online store and make purchases from their mobile devices, or a photo-sharing app may allow users to take photos, edit them, and then store them in the cloud for sharing. Like fully native apps, hybrid apps are distributed to users through an application store. Tools such as Amazon Web Services (AWS), Mobile Hub, or Google Firebase support development of both hybrid and mobile web apps.
Why do mobile application testing?
An app that doesn’t meet Apple’s standards for quality won’t even make it past the review process and into the application store. Mobile app users are quick to delete applications that freeze, crash, are slow to load, or have a poor user experience – and leave bad reviews in the application store. With tens of thousands of apps available to users, even a few bad reviews can be the death knell for an application. Once a defect in a native or hybrid app has made its way onto a mobile device, fixing it is a time-consuming effort that includes re-submitting the app to the application store, waiting for review by Apple and waiting for the user to download the update.
Effective testing is critical to minimize defects and improve quality in mobile applications, leading to apps that keep users engaged and earn high ratings from users. For this reason, mobile application tests must be well-designed to cover not just the functionality of the app, but also its ease of use and performance. Users won’t tolerate apps that use unreasonable amounts of data or that drain their device’s battery. The app must also be able to respond gracefully to events such as turning the device off, putting it into airplane mode, or connecting to another device via WiFi, USB or Bluetooth.
Device variety
One of the biggest challenges in mobile application testing is the wide variety of mobile devices available, including tablets, e-readers, mobile phones, and wearables. These devices have different input methods, such as physical keyboards, virtual keyboards, touch screens, trackpads, buttons, and switches. They also have features not found on desktop or laptop computers, such as GPS receivers, gyroscopes, heart rate monitors, and fingerprint or face scanners. Mobile devices offer different screen sizes and resolutions, and may be used in landscape or portrait screen orientation.
O/S variety
Mobile devices have different operating systems, and different versions of the same operating system are typically in use at any given point in time. In just the Apple ecosystem, iPads and iPhones may be using iOS 10 or 11, while Apple TVs use tvOS and Apple watches use watchOS. Providers that offer streaming media might have to develop versions of their app for all three Apple operating systems – and that doesn’t include all of the versions of the Android operating system available. According to Software Testing News, in 2018, a new manual testing lab would need almost 50 devices just to provide 80% coverage of the possible combinations.
Connectivity
Other considerations for mobile testing include the variety of connectivity options such as Bluetooth, WiFi or mobile network. For both WiFi and mobile networks, applications must be able to respond well to a variety of connection speeds, such as Edge, 3G or LTE. Also important for testing is the response of an application to loss of connectivity – for example, when a user puts a device in airplane mode or moves into an area without cell coverage.
Continuous Testing
Mobile device users expect apps to update frequently – both to add features and to maintain compatibility with frequent O/S releases. To meet this expectation, many mobile development teams use agile development methods, including continuous integration and continuous deployment. In turn, this creates an environment of continuous testing, where fast feedback is essential.
Looking deeper: O/S variety
With Microsoft ending support for Windows Phone 8.1 and plans for Windows 10 Mobile uncertain, the two major players remaining in the mobile O/S market are Google Android and Apple iOS. The following statistics from Software Testing News, published in February 2017, show the market share between the two rival ecosystems.
Smartphone Market Share
Android is the leader with about 85% of the smartphone market, while iOS has 15%.
Mobile Market Share
When tablets are included, Android has about 65% of the total mobile market, with iOS at 33%.
Android Versions in Use
Worldwide, there are more than two billion active Android devices in use, but significant percentages are not on the latest version. This is because while Apple controls what version of its OS is sold with new devices, Google does not, but instead relies on device vendors to update their new phones and tablets. The result is a fragmented install base, which creates a significant challenge for application testing. As of August 2017, 15% of Android devices are running version 4.4 KitKat, 29% are running 5.0 or 5.1 Lollipop, 32% are running 6.0 Marshmallow, and 16% are running 7.0 or 7.1 Nougat, with the remainder running versions prior to 4.4 KitKat.
Mobile application testing levels: when to test
To produce the highest quality, it is important to test throughout the entire development process, beginning with the design phase and continuing through delivery to the application store.
Design testing
Test your mobile app’s design to ensure that the application has the right functionality and ease-of-use to keep users engaged. For example, the content should be responsive to the size and orientation of the device so that users do not have to scroll, and the app should respond appropriately to touch controls. Apple offers detailed guidelines for UI design including specific suggestions for button and text size, use of progress indicators and alerts, application settings, and controls. Likewise, Google offers a wealth of design resources for Android developers. These resources are extremely beneficial for QA personnel responsible for reviewing the design and usability of mobile applications.
A common approach to improving the quality of application design is A/B testing, which compares two designs to see which provides the better user experience (UX). In A/B testing, two groups of users each evaluate a feature of an app, such as its menus or registration process. Testers observe the users
Unit testing
Unit testing is performed by developers on individual units of code, prior to merging the new code into the application. The rationale behind unit testing is to reduce the cost of finding and correcting defects – a cost which increases the later in the development process a defect is identified. In continuous integration environments, unit tests are completed by developers prior to committing their code to the shared repository. In mobile app testing, unit testing requires using the API calls supported by mobile development environments. Android is integrated with the JUnit framework for automating unit testing. For iOS, XCTest is provided as part of the iOS SDK and works in the Xcode development environment.
Integration testing
Integration testing verifies that new code works properly when integrated into an application. When several developers are working on the same project, integration testing helps ensure that their code changes do not conflict with each other. To make this process more efficient, many mobile development teams work in a continuous integration environment, where developers “check-in” code changes to a shared repository at least daily, if not more often. A continuous integration server such as Jenkins, GitLab or Bamboo can build the application with the new code and perform integration testing automatically.
Acceptance testing
Acceptance testing verifies that a mobile application provides the functionality, usability, and performance to meet the user’s needs, and that it delivers a consistent user experience (UX) across platforms. An important part of acceptance testing is beta testing on real devices by real end-users. The Apple Developer Program offers the free TestFlight tool for managing beta testing. TestFlight currently supports up to 25 internal testers and 10,000 external testers, who are invited to participate in beta testing by their email address. Likewise, the Google Play Store has a developer console for beta testing with distribution to testers through Google Groups or Google+ communities. There are also third-party options available for managing beta testing in both iOS and Android, such as the popular TestFairy.
Mobile application testing types: what to test
Several types of testing occur during mobile application development, including functional testing, non-functional testing, and regression testing.
Functional testing
Functional testing investigates whether an application does what it is supposed to do, usually by comparing it to design documents. It is used to examine the user interface including UI layouts and elements, menu structure and functions, as well as the response to screen orientation and screen resolution changes. For native apps, functional testing includes ensuring that the installation process completes successfully as well as testing any registration or activation process that the user must complete. You must also test the update process to ensure that no data is lost or corrupted during an update. Localization settings, if used, must also be verified. Functional testing also verifies that the app supports accessibility features for users with vision loss, hearing loss, or other disabilities. Mobile web applications must be tested for compatibility with each mobile operating system as well as the common browser on each O/S. For hybrid applications, it is also important to test backend services, mobile back-end as a service (Mbaas), Baas, Backend Services.
Non-functional testing
Non-functional testing includes operational testing, interrupt testing, security testing, and performance testing.
Operational testing
Operational testing includes checking the behavior of the AUT during events such as cycling the power on/off, insertion or removal of a data cable, turning WiFi on or off, and entering airplane mode.
Interrupt testing
Mobile applications are frequently interrupted by events on the device. Interrupt testing verifies that the AUT can handle these events appropriately. Interrupt testing includes events such as making or receiving phone calls, sending and receiving SMS/MMS messages, and pop-up notifications from the O/S such as low battery indicators and alarm reminders.
Security testing
Security testing is especially important in today’s environment. Mobile devices contain highly sensitive user data, including fingerprints or face scans, location data, private photos, payment information, browsing history, personal communications, and more. Users are concerned about how companies are using this data and shielding it from malicious activity. Mobile apps may store sensitive data in the cloud; mobile application shielding of sensitive data must be tested thoroughly.
Performance testing
Performance testing is critical in mobile environments, as users will be quick to delete applications that run slowly, consume too much battery or data, or consume memory which can lead to unexpected shutdowns. The list below provides a checklist of items for performance testing:
- Application speed, including how quickly the application opens, how quickly it retrieves or stores information from back-end services, and whether there are any unexpected lags in functionality
- Battery usage, including not only how much battery the mobile application demands but also how the application performs in a low-battery situation
- Data consumption demands during typical use scenarios, as well as in edge cases of minimal and maximum use
- Low-level resource/memory usage/memory leakage. Memory leaks occur when an application feature fails to release memory when it is no longer needed, which can lead to application crashes.
Regression testing
Regression testing involves re-running tests that were successful in previous versions of the mobile application. This is done to ensure that new code hasn’t re-introduced old defects, called “regressions.” Because regression tests are performed over and over again, they are ideal candidates for automation. To learn more about regression testing, refer to our Regression Testing Guide.
Mobile testing techniques: how to test
Mobile application testing can be either manual or automated, and performed on simulators, emulators or physical devices. Manual testing on physical devices provides the most realistic recreation of a user’s environment but can be time-consuming and expensive. Automated mobile testing can improve testing efficiency to minimize the time to release. Once a test has been automated, it can be performed again with each new version of an application, to ensure that new code does not introduce regressions.
Simulators reproduce the software environment of an app and are suitable for preliminary testing and sanity testing. Apple provides a simulator as part of its Xcode developer toolkit. One drawback of simulators is that they don’t always behave like real devices, especially in the area of low-level resources like memory and battery. Emulators go one step further and attempt to reproduce the entire environment of the device including hardware. The Android Studio includes an Android emulator that “provides almost all the capabilities of a real Android device. You can simulate incoming phone calls and text messages, specify the location of the device, simulate different network speeds, simulate rotation and other hardware sensors, access the Google Play Store, and much more.”
If possible, test your application first on simulators or emulators, to eliminate as many defects as possible prior to going through the installation process on physical devices. Begin testing with smoke and sanity tests to verify that basic functionality is in place before committing valuable time and resources to further testing.
Physical devices provide the most realistic environment for testing your application. It is especially important to conduct performance testing on physical devices to ensure that users will be satisfied with your application’s speed, ease-of-use, and consumption of low-level resources.
Provisioning of Apple devices
Native or hybrid apps for Apple devices must be provisioned before they can be tested on physical devices. Provisioning is the process of preparing and configuring an app to launch on devices and use app services, which includes installation of the app, configuration of user accounts, and code signing of the app. Provisioning is accomplished in the Apple Xcode development environment. No provisioning is necessary for mobile web applications, as nothing is installed on the target device.
On Apple iOS and tvOS devices, provisioning begins with registering test devices in an Apple developer account. The next step is creating an iOS application archive file (.ipa) containing the app and creating an ad hoc provisioning profile, which supports the installation of the beta version of the app on designated devices. The profile includes the ID of the application, the unique device ID (UDID) of the test devices, and a distribution certificate. Once these exist, the app can be installed by connecting the iOS or tvOS devices directly to a Mac, or using iTunes for iOS and watchOS devices. Apple Configurator can also be used to install an app on all three platforms.
Preparation of Android devices
On Android devices, the test preparation process begins with putting the Android device into “Developer Mode” and enabling USB debugging. You must also enable debugging in the application itself. Then, connect the device to a Mac or Windows computer with a USB cable and use the Android development environment to load the app onto the device.
Considerations in selecting mobile testing tools
Automating mobile tests is essential to provide the fast feedback needed in continuous testing environments. There are both open source and proprietary solutions available. Appium is a leading open source tool for testing native, mobile web and hybrid applications on iOS, Android and Windows. Appium uses the popular, open-source Selenium WebDriver protocol.
When selecting a tool there are several aspects to consider:
- Can the automation tool execute tests in parallel on multiple devices?
- Does it support both simulators/emulators and real devices?
- Does it offer reusable code modules and provide data-driven testing?
- Does it offer support for native apps, mobile web apps, and hybrid apps for all major platforms?
- Does the tool have a design once, run everywhere approach with unique configurations for each endpoint?
- Does the tool support leading open source solutions including Selenium Webdriver nodes/grids and Appium?
- Does the tool support JSON capabilities for instrumenting hybrid and mobile apps?
- Does the tool offer GUI object recognition or is it limited to image recognition?
Tip: If you are ready to start automating your mobile application testing, download the free trial of Ranorex Studio today. Or, contact one of our test automation experts.
Related Posts:
Model-Based Testing with Ranorex DesignWise
Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
What Is OCR (Optical Character Recognition)?
Optical character recognition technology (OCR), or text recognition, converts text images into a machine-readable format. In an age of growing need for efficient data extraction and analysis processes, OCR has helped organizations revolutionize how they process and...
Support Corner: API Testing and Simple POST Requests
Ranorex Studio is renowned for its robust no-code capabilities, which allow tests to be automated seamlessly across web, mobile, and desktop applications. Beyond its intuitive recording features, Ranorex Studio allows custom code module creation using C# or VB.NET,...