Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
Test Automation Guide
This guide covers key concepts in test automation, including the types of software tests that can be automated, popular tools for automating testing, and strategies to get started with test automation.
What is test automation?
Test automation is simply the practice of using software to test software. An automated test case consists of a series of commands that execute the steps of a test, as well as any data needed for the test and the expected result. For example, instead of a person manually testing a login process by entering a series of usernames and passwords, an automated test case submits the usernames and passwords to the application under test (AUT), compares the expected results to the actual results to determine whether or not the test passed or failed, and creates a report summarizing the results of the test
An automated test case, then, consists of a series of commands that execute the steps of a test, as well as any data needed for the test and the expected result. For example, instead of a person manually testing a login process by entering a series of usernames and passwords, an automated test case submits the usernames and passwords to the application under test (AUT), compares the expected results to the actual results to determine whether or not the test passed or failed, and creates a report summarizing the results of the test.
Automation is ideal for test cases that will be run multiple times, including the following:
- Tests that need to be repeated for multiple web browsers (i.e. cross-browser tests) or mobile devices
- Tests that are part of a regression suite
- Are executed as part of a continuous integration (CI) process
In theory, any test that can be done manually can be automated, although there are a number of factors to consider in determining whether automating a particular test case is beneficial.
One strategic consideration of automation is that it does more than just reduce the labor of particular tests: it changes when and how often tests are run. Suppose a test is long and complicated and requires a lot of resources; humans might delicately schedule such a test to run once a quarter. When that same test is automated, though, it can be run every night, during hours the datacenter is lightly loaded. To receive test results two orders of magnitude faster utterly transforms the development process and its outcomes. Test automation, then, can yield not just cost savings, but qualitatively better processes and results.
Test automation increases efficiency when a test case must be run multiple times, such as testing a web application for cross-browser compatibility
Watch our on-demand webinar
Strategies for a Successful Automation Project: Learn how to ensure that your automation project accomplishes your goals.
Why use test automation?
Manual testing requires significant resources, including time, personnel, and system resources such as network bandwidth, servers, and database storage. In fast-paced development environments, there may simply not be enough time available for comprehensive manual testing, leaving teams with the choice of either delaying software delivery or releasing without adequate testing. There is always the possibility with manual testing that scripted tests are not performed exactly as specified, or that exploratory tests are not adequately documented, making errors potentially unreproducible and therefore difficult to debug. Automated testing offers potential solutions to the drawbacks of manual testing.
Benefits of automated testing
Fast feedback
The key benefit of automated testing is the ability to execute more tests in less time, thus providing valuable feedback to development teams to help determine whether software is ready for release. This feedback begins early in the software development life cycle through automated unit tests, when the cost to resolve a defect is relatively low.
Repeatability
Automated tests are also more consistent and less expensive to repeat than manual tests. For example, automation testing makes it possible to execute the same test case in exactly the same way for multiple data values, or across multiple operating systems or web browsers. This frees up human testers to conduct exploratory testing as well as execute test cases that would be too expensive to automate. And, once a test is automated, it can be re-used for multiple purposes.
Time and cost savings
Automation can greatly reduce the time and cost of software testing. In continuous integration environments, automated regression test cases can be triggered automatically for code changes as soon they are checked into the repository. Automated interface/API tests and system-level tests can run overnight and in parallel when system resources are more available. In summary, automation makes it possible to run more tests more often to increase code coverage, increasing confidence that the application is ready for release.
Quality
Automated testing not only helps increase test coverage, but also frees test personnel to perform more challenging and exploratory testing. In addition, test-first development approaches such as acceptance test-driven development (ATDD) and behavior-driven development (BDD) help ensure that an application has the right functionality to meet customer requirements. Earlier testing combined with increased test coverage improves confidence in the quality of software released.
This section has already mentioned twice automated testing’s potential to “free test personnel”. Think of this as a division of labor, with all the payoffs that arrangement brings: the computer runs tests that are inconvenient, tedious, boring, and hard to get right, while humans concentrate on creative and exploratory testing. This kind of teamwork can multiply testing effectiveness.
Manageability
Discussions of testing techniques generally focus on “the left side”, where we detect and diagnose errors. A more comprehensive testing framework generally also helps report those errors in a consistent, useful fashion. This makes it easier to think about and act effectively on aggregated test results. Rather than addressing each error in isolation, good automation tooling helps us manage the entire body of errors to achieve the greatest improvement in quality with the least effort.
Challenges of automated testing
Startup cost
Startup costs for test automation can include the test software and hardware, as well as costs to train existing staff or hire experts that are adept in automation tools. Once the testing resources are in place, there are additional costs in time and resources to create, execute, and maintain automated tests. Generally, it is more expensive to automate any given individual test case than to execute it one time, manually.
Selecting the right tool
Many tools are available for building automated tests, including both “free” open-source tools and paid, proprietary solutions. Most development and QA teams employ a combination of tools to build a solution with the necessary testing functionality for their unique environment, rather than using a single tool
Maintenance
Once automated tests have been created, they often require on-going maintenance to continue to work properly when the AUT changes. Best practices in test automation can help keep these maintenance costs to a minimum.
What is a test automation framework?
A test automation framework is a set of rules, libraries and functions that guide automation of a test project. A complete test automation framework will typically include a repository of automated test cases, test data, configuration information such as test setup/teardown, test management tools, and libraries. The following are common types of automation frameworks:
Behavior-driven
Component testing checks individual units of code. Component testing is often called unit testing, but may also be called module testing or program testing. Developers write and execute unit tests to find and fix defects in their code as early as possible in the development process. This is critical in agile development environments, where short release cycles require fast test feedback. Unit tests are white-box tests because they are written with a knowledge of the code being checked.
Modular
In a modular framework, automated tests are built as individual modules that can be used by other modules. For example, a test case to validate a login function may call individual modules that start the application, enter and validate a username and password combination, shut down the application, and report the results. The benefit of this approach is that if the requirements for a valid password change, only that one module needs to be updated; the other modules can continue to be used without any change.
Data-driven
In a data-driven framework, a given test case is executed once for each record in a data table. In the example of a test case to validate a login, you would first create a data table, such as an Excel file, with a series of usernames and passwords to be tested. The framework would then repeat the login test once for each row in the Excel table. Because the test data is separate from the test case, it is easy to add or change the values to be tested if the requirements for a password are changed.
Keyword-driven
Like data-driven testing, keyword-driven testing uses a table to drive testing. But instead of data, the controlling table contains a list of actions for the test to perform. The open-source Robot framework is an example of a keyword-driven approach.
Hybrid
A hybrid framework combines two or more of the framework types described above.
Ranorex Studio is a full test automation framework that supports modular, data-driven, and keyword-driven testing of desktop, web-based and mobile applications written in almost any language.
How test automation works in Ranorex Studio
The following section describes the process of automating a GUI test using Ranorex Studio tools.
Identify UI elements: The first step in test automation is to identify the objects that make up the user interface, including text fields and controls such as forms, buttons, drop-down menus and checkboxes. Ranorex Studio does this using the Ranorex Spy tool, which identifies each element using its RanoreXPath. The RanoreXPath syntax is based on XPath, a W3C web development standard that can be used to navigate through elements in an XML document. However, the RanoreXPath is not limited to web applications, but can also identify UI elements in desktop and mobile applications. Ranorex Spy generates more stable object identification, even for dynamic objects, and is less subject to failure simply because an element changes location on the screen.
Manage UI elements: Once a UI element is identified, it is added to the object repository, which uses a hierarchical tree structure to organize UI objects. The repository separates the identification of an object from automated tests, to improve maintainability. Each repository object has a unique, meaningful name, and can also be linked to variables for parameter-driven or data-driven testing.
Create automation modules: The next step is to create automation modules, which identify the actions that the test should perform, the data values to use for the test, and criteria for determining whether or not the test was successful. Automation modules can be created without coding using Ranorex Studio’s capture-and-replay feature, which identifies UI elements, adds them to the repository, and creates an editable automation module – all in a single recording session. Or, the steps that the test module should perform can be written in C# or VB.NET code. In addition to modules that automate individual test cases, Ranorex Studio also supports automation modules that set up and teardown the test environment, execute other automation modules based on conditions, and more. A single test project may have dozens of automation modules.
Capture-and-replay functionality supports creation of test automation modules without coding.
What are the leading test automation tools?
Test automation can contribute to every stage of the software development life cycle. Below are some of the leading automation tools.
TDD/ATDD/BDD tools
These tools support quality in software development through a test-first approach such as test-driven design (TDD), acceptance-test driven design (ATDD) or behavior-driven design (BDD). In BDD environments, user requirements are written as statements in the Gherkin language, using the “given-when-then” syntax. Gherkin statements can be executed as acceptance tests by a tool such as Cucumber. Cucumber was originally developed for applications written in the Ruby programming language, but it has been implemented for other languages as well, including SpecFlow for .Net applications, Cucumber-jvm and JBehave for Java applications, and Cucumber.js for JavaScript applications.
Unit testing tools
Automated unit tests require a tool that can mock missing services, such as a database server, or provide a “stub” to replace missing integrations. Popular unit testing tools include JUnit and TestNG for Java applications, Jasmine for JavaScript applications, and nUnit or Microsoft Visual Studio Unit Testing Framework for .Net applications. For iOS devices, XCTest is provided as part of the Apple Xcode development platform.
Web and mobile testing frameworks
Selenium WebDriver is the leading open-source automation framework for web applications. The Selenium framework includes a server, client drivers for various programming languages, an integrated development environment (IDE), and third-party plugins with additional features. Appium automates testing for native, web and hybrid mobile apps for both Android and iOS devices, and supports XCTest on iOS.
Continuous integration tools
Continuous integration is a development practice that emphasizes frequent builds of an application, with the goal of identifying and resolving defects immediately. A continuous integration server can execute regression test scripts automatically for events such as code commits. Leading open-source continuous integration tools include Jenkins and Hudson, while Bamboo is a popular proprietary tool.
Test coverage analysis tools
Test coverage analysis tools can identify how much of an AUT is verified by automated tests. EMMA is a leading test coverage analysis tool for Java, while dotCover is a plug-in to Microsoft Visual Studio that automates test coverage analysis for .NET applications.
Defect reporting and issue tracking
These tools help testers and developers communicate about defects and track them through reporting to resolution. Leading tools include Atlassian JIRA and Bugzilla.
Performance testing
These tools help testers and developers communicate about defects and track them through reporting to resolution. Leading tools include Atlassian Jira and Bugzilla.
Source control
These tools help manage the source code of an application, which is essential when a team of developers is working on the same application. Typically, developers check out a module from a repository, and then commit code changes back to the repository when finished. A source control tool can also manage the source code for automated tests, as well as the application under test. Git is one leading open-source solution. GitHub is a web-hosted version of Git.
Getting started: the test automation process
Test automation should be approached as a software development process in its own right.
Key steps in the test automation process include the following:
- Choose a testing framework approach and tools. Download open-source tools and free trials of proprietary tools to explore their features. Ranorex offers a free trial to help determine whether Ranorex Studio is the right fit for your testing needs.
- Treat test automation as a software development project in its own right. For example, set naming standards and other standards such as designating a Git repository for automated test cases.
- Carefully select GUI test cases for automation, prioritizing those that will be used for regression testing or cross-browser testing, smoke tests that will be executed by a CI server, or test cases that will be executed for a set of data values.
- Build tests to be as modular and re-usable as possible to reduce the cost of maintenance. For example, don’t include logon steps in each test case. Rather, write a logon script in its own module, and have the other test cases call the logon module. If the logon process changes, only that one test case will need to be updated.
- Put in place a process to archive test cases that are no longer needed; see the regression testing wiki article for more information.
A continuous integration server executes automated regression tests to ensure that each build of an application meets quality standards
Developing a test automation strategy
To help ensure that the costs of automation do not outweigh the benefits, a team’s automation efforts should focus on areas that are likely to have a higher return on investment, such as test cases that will be reused for regression testing or cross-browser testing.
A well-known recommendation for managing the cost/benefit ratio is given in the test automation pyramid below, originally developed by Mike Cohn, but extended and adapted by other leaders in the software testing industry. According to the pyramid, the greatest number of automated tests should be unit tests, since they are relatively inexpensive to develop and provide feedback early in development. The test automation pyramid is sometimes interpreted as suggesting that user interface testing is unimportant, or that GUI tests shouldn’t be automated, but neither is true. GUI testing is essential for evaluating an application as a whole, from the business and/or user perspective. By validating the visual and functional aspects of an interface as well as user workflows, GUI testing helps ensure a positive user experience. Manual session-based exploratory testing sits atop the automation pyramid, complementing the automated testing.
The test automation pyramid model suggests that the greatest number of automated tests should be at the unit testing level.
How to learn automation testing
Based on a review of over 150 job postings for software testers recently conducted by Ranorex, a majority of software testing jobs call for test automation skills. The best way to get started in test automation is to learn by doing: Download the free trial of Ranorex Studio, and complete the tutorial in the first section of our online user guide. After completing this tutorial, Ranorex provides additional free resources to help you get started in test automation, including our webinars, which are available both live and on-demand, and our series of screencasts.
If you would like to see how test automation can help you create and execute GUI test cases, and report the results of GUI testing, download a trial version of the Ranorex test automation solution. Or, contact one of our test automation experts.
At some point in your software QA career, you’ll want to learn how such professional organizations as the Association for Software Testing, the International Software Testing Qualifications Board, the American Productivity and Quality Center, and the American Society for Quality can help you.
Most of the previous resources assume at least rudimentary exposure to programming languages. Whatever your starting position, you’ll do well to read “Why it’s (still) so difficult for testers to learn test automation” and “7 Quick Steps to Become a Great Automation Testing Engineer”. In any case, don’t let inexperience or lack of previous training discourage you; the demand for enthusiastic, capable test automators is large, and even a few weeks of concentrated exercise as outlined above will make you a valuable practitioner.
Related Posts:
Model-Based Testing with Ranorex DesignWise
Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
What Is OCR (Optical Character Recognition)?
Optical character recognition technology (OCR), or text recognition, converts text images into a machine-readable format. In an age of growing need for efficient data extraction and analysis processes, OCR has helped organizations revolutionize how they process and...
Support Corner: API Testing and Simple POST Requests
Ranorex Studio is renowned for its robust no-code capabilities, which allow tests to be automated seamlessly across web, mobile, and desktop applications. Beyond its intuitive recording features, Ranorex Studio allows custom code module creation using C# or VB.NET,...