Share this article:

Test Automation Guide

This guide covers key concepts in test automation, including the types of software tests that can be automated, popular tools for automating testing, and strategies to get started with test automation.

What is test automation?

Test automation is simply the practice of using software to test software. An automated test case consists of a series of commands that execute the steps of a test, as well as any data needed for the test and the expected result. For example, instead of a person manually testing a login process by entering a series of usernames and passwords, an automated test case submits the usernames and passwords to the application under test (AUT), compares the expected results to the actual results to determine whether or not the test passed or failed, and creates a report summarizing the results of the test. Automation is ideal for test cases that will be run multiple times, such as tests that need to be repeated for multiple web browsers or mobile devices, those that are part of a regression suite, and those that are executed as part of a continuous integration process. In theory, any test that can be done manually can be automated, although there are a number of factors to consider in determining whether automating a particular test case is beneficial.

Cross Browser compatibility testing Test automation increases efficiency when a test case must be run multiple times, such as testing a web application for cross-browser compatibility

Why use test automation?

Manual testing requires significant resources, including time, personnel, and system resources such as network bandwidth, servers, and database storage. In fast-paced development environments, there may simply not be enough time available for comprehensive manual testing, leaving teams with the choice of either delaying software delivery or releasing without adequate testing. There is always the possibility with manual testing that scripted tests are not performed exactly as specified, or that exploratory tests are not adequately documented, making errors potentially unreproducible and therefore difficult to debug. Automated testing offers potential solutions to the drawbacks of manual testing.

Benefits of automated testing

Fast feedback

The key benefit of automated testing is the ability to execute more tests in less time, thus providing valuable feedback to development teams to help determine whether software is ready for release. This feedback can begin early in the software development life cycle through automated unit tests, when the cost to resolve a defect is relatively low.

Repeatability

Automated tests are also more consistent and less expensive to repeat than manual tests. For example, automation testing makes it possible to execute the same test case in exactly the same way for multiple data values, or across multiple operating systems or web browsers. This frees up human testers to conduct exploratory testing as well as execute test cases that would be too expensive to automate. And, once a test is automated, it can be re-used for multiple purposes.

Time and cost savings

Automation can greatly reduce the time and cost of software testing. In continuous integration environments, automated regression test cases can be triggered automatically for code changes as soon they are checked into the repository. Automated interface/API tests and system-level tests can run overnight and in parallel when system resources are more available. In summary, automation makes it possible to run more tests more often to increase code coverage, increasing confidence that the application is ready for release.

Quality

Automated testing allows more tests to be executed for an application in less time, not only increasing test coverage, but also freeing test personnel to perform more challenging and exploratory testing. In addition, test-first development approaches such as acceptance test-driven development (ATDD) and behavior-driven development (BDD) help ensure that an application has the right functionality to meet customer requirements. Earlier testing combined with increased test coverage improves confidence in the quality of software released.

Challenges of automated testing

Startup cost

Startup costs for test automation can include the test software and hardware, as well as costs to train existing staff or hire experts that are adept in automation tools. Once the testing resources are in place, there are additional costs in time and resources to create, execute, and maintain automated tests. Generally, it is more expensive to automate any given individual test case than to execute it one time, manually.

Selecting the right tool

Selecting the right tool there are many tools available for building automated tests, including both “free” open-source tools and paid, proprietary solutions. Most development and QA teams employ a combination of tools to build a solution with the necessary testing functionality for their unique environment, rather than using a single tool

Maintenance

Once automated tests have been created, they often require on-going maintenance to continue to work properly when the AUT changes. Best practices in test automation can help keep these maintenance costs to a minimum.

What is a test automation framework?

A test automation framework is a set of rules, libraries and functions that guide automation of a test project. A complete test automation framework will typically include a repository of automated test cases, test data, configuration information such as test setup/teardown, test management tools, and libraries. The following are common types of automation frameworks:

Behavior-driven

A behavior-driven framework creates test cases based on specifications that are defined as user behaviors, often using the “given-when-then” format. A BDD statement for a login might read something like “Given the user is on the login screen, when the user enters a valid username and password, and clicks Login, then the user sees the home page. A BDD framework converts BDD statements into executable test cases.

Modular

In a modular framework, automated tests are built as individual modules that can be used by other modules. For example, a test case to validate a logon function may call individual modules that start the application, enter and validate a username and password combination, shut down the application, and report the results. The benefit of this approach is that if the requirements for a valid password change, only that one module needs to be updated; the other modules can continue to be used without any change.

Data-driven

In a data-driven framework, a given test case is executed once for each record in a data table. In the example of a test case to validate a login, you would first create a data table, such as an Excel file, with a series of usernames and passwords to be tested. The framework would then repeat the login test once for each row in the Excel table. Because the test data is separate from the test case, it is easy to add or change the values to be tested if the requirements for a password are changed.

Keyword-driven

Like data-driven testing, keyword-driven testing uses a table to drive testing. But instead of data, the controlling table contains a list of actions for the test to perform. The open-source Robot framework is an example of a keyword-driven approach.

Hybrid

A hybrid framework combines two or more of the framework types described above.

Ranorex Studio is a full test automation framework that supports modular, data-driven, and keyword-driven testing of desktop, web-based and mobile applications written in almost any language.

How test automation works in Ranorex Studio

The following section describes the process of automating a GUI test using Ranorex Studio tools.

Automation of GUI test

Identify UI elements: The first step in test automation is to identify the objects that make up the user interface, including text fields and controls such as forms, buttons, drop-down menus and checkboxes. Ranorex Studio does this using the Ranorex Spy tool, which identifies each element using its RanoreXPath. The RanoreXPath syntax is based on XPath, a W3C web development standard that can be used to navigate through elements in an XML document. However, the RanoreXPath is not limited to web applications, but can also identify UI elements in desktop and mobile applications. Ranorex Spy generates more stable object identification, even for dynamic objects, and is less subject to failure simply because an element changes location on the screen.

Manage UI elements: Once a UI element is identified, it is added to the object repository, which uses a hierarchical tree structure to organize UI objects. The repository separates the identification of an object from automated tests, to improve maintainability. Each repository object has a unique, meaningful name, and can also be linked to variables for parameter-driven or data-driven testing.

Create automation modules: The next step is to create automation modules, which identify the actions that the test should perform, the data values to use for the test, and criteria for determining whether or not the test was successful. Automation modules can be created without coding using Ranorex Studio’s capture-and-replay feature, which identifies UI elements, adds them to the repository, and creates an editable automation module – all in a single recording session. Or, the steps that the test module should perform can be written in C# or VB.NET code. In addition to modules that automate individual test cases, Ranorex Studio also supports automation modules that set up and teardown the test environment, execute other automation modules based on conditions, and more. A single test project may have dozens of automation modules.

Ranorex GUI object recognition Capture-and-replay functionality supports creation of test automation modules without coding.

Execute tests: Tests can be run from within the Ranorex Studio desktop interface. Because Ranorex Studio saves test projects as executable .EXE files, tests can also be run from the command line without using the desktop interface, and can be integrated into a number of other processes, such as continuous integration. Ranorex Studio generates easily-readable test execution reports to help teams rapidly identify and resolve defects.

What are the leading test automation tools?


Test automation can contribute to every stage of the software development life cycle. Below are some of the leading automation tools.

TDD/ATDD/BDD tools

These tools support quality in software development through a test-first approach such as test-driven design (TDD), acceptance-test driven design (ATDD) or behavior-driven design (BDD). In BDD environments, user requirements are written as statements in the Gherkin language, using the “given-when-then” syntax. Gherkin statements can be executed as acceptance tests by a tool such as Cucumber. Cucumber was originally developed for applications written in the Ruby programming language, but it has been implemented for other languages as well, including SpecFlow for .Net applications, Cucumber-jvm and JBehave for Java applications, and Cucumber.js for JavaScript applications.

Unit testing tools

Automated unit tests require a tool that can mock missing services, such as a database server, or provide a “stub” to replace missing integrations. Popular unit testing tools include JUnit and TestNG for Java applications, Jasmine for JavaScript applications, and nUnit or Microsoft Visual Studio Unit Testing Framework for .Net applications. For iOS devices, XCTest is provided as part of the Apple Xcode development platform.

Web and mobile testing frameworks

Selenium WebDriver is the leading open-source automation framework for web applications. The Selenium framework includes a server, client drivers for various programming languages, an integrated development environment (IDE), and third-party plugins with additional features. Appium automates testing for native, web and hybrid mobile apps for both Android and iOS devices, and supports XCTest on iOS.

Continuous integration tools

Continuous integration is a development practice that emphasizes frequent builds of an application, with the goal of identifying and resolving defects immediately. A continuous integration server can execute regression test scripts automatically for events such as code commits. Leading open-source continuous integration tools include Jenkins and Hudson, while Bamboo is a popular proprietary tool.

Test coverage analysis tools

Test coverage analysis tools can identify how much of an AUT is verified by automated tests. EMMA is a leading test coverage analysis tool for Java, while dotCover is a plug-in to Microsoft Visual Studio that automates test coverage analysis for .NET applications.

Test Management Software

Test management tools such as TestRail help track the progress of testing sessions, providing reports and graphs to help analyze both automated and manual testing efforts.

Defect reporting and issue tracking

These tools help testers and developers communicate about defects and track them through reporting to resolution. Leading tools include Atlassian JIRA and Bugzilla.

Performance testing

Performance testing tools automate the process of conducting load and stress testing, to ensure that an application can support anticipated user loads. Popular tools include NeoLoad and Apache Jmeter.

Source control

These tools help manage the source code of an application, which is essential when a team of developers is working on the same application. Typically, developers check out a module from a repository, and then commit code changes back to the repository when finished. A source control tool can also manage the source code for automated tests, as well as the application under test. Git is the leading open-source solution. GitHub is a web-hosted version of Git.

Getting started: the test automation process

Test automation should be approached as a software development process in its own right.

Key steps in the test automation process include the following:

  • Choose a testing framework approach and tools. Download open-source tools and free trials of proprietary tools to explore their features. Ranorex offers a free evaluation webinar to help determine whether Ranorex Studio is the right fit for your testing needs.
  • Treat test automation as a software development project in its own right. For example, set naming standards and other standards such as designating a Git repository for automated test cases.
  • Carefully select GUI test cases for automation, prioritizing those that will be used for regression testing or cross-browser testing, smoke tests that will be executed by a CI server, or test cases that will be executed for a set of data values.
  • Build tests to be as modular and re-usable as possible to reduce the cost of maintenance. For example, don’t include logon steps in each test case. Rather, write a logon script in its own module, and have the other test cases call the logon module. If the logon process changes, only that one test case will need to be updated.
  • Put in place a process to archive test cases that are no longer needed; see the regression testing wiki article for more information.
Automated Regression test A continuous integration server executes automated regression tests to ensure that each build of an application meets quality standards

Developing a test automation strategy

To help ensure that the costs of automation do not outweigh the benefits, a team’s automation efforts should focus on areas that are likely to have a higher return on investment, such as test cases that will be reused for regression testing or cross-browser testing.

A well-known recommendation for managing the cost/benefit ratio is given in the test automation pyramid below, originally developed by Mike Cohn, but extended and adapted by other leaders in the software testing industry. According to the pyramid, the greatest number of automated tests should be unit tests, since they are relatively inexpensive to develop and provide feedback early in development. The test automation pyramid is sometimes interpreted as suggesting that user interface testing is unimportant, or that GUI tests shouldn’t be automated, but neither is true. GUI testing is essential for evaluating an application as a whole, from the business and/or user perspective. By validating the visual and functional aspects of an interface as well as user workflows, GUI testing helps ensure a positive user experience. Manual session-based exploratory testing sits atop the automation pyramid, complementing the automated testing.

Early GUI - Apple Mcintosh The test automation pyramid model suggests that the greatest number of automated tests should be at the unit testing level.

How to learn automation testing

Based on a review of over 150 job postings for software testers recently conducted by Ranorex, a majority of software testing jobs call for test automation skills. The best way to get started in test automation is to learn by doing: Download the free trial of Ranorex Studio, and complete the online tutorial which makes up the first 9 chapters in the online user guide. After completing this tutorial, Ranorex provides additional free resources to help you get started in test automation, including our webinars, which are available both live and on-demand, and our series of screencasts.

If you would like to see how test automation can help you create and execute GUI test cases, and report the results of GUI testing, download a trial version of the Ranorex test automation solution. Or, contact one of our test automation experts at [email protected].

Enjoy this article? Don’t forget to share!

Association for Software Testing: https://associationforsoftwaretesting.org

International Software Testing Qualifications Board: http://www.istqb.org

Ask a Ranorex expert

Get your most pressing questions answered by one of our Ranorex experts.

Simply fill out the form below

Ask an expert