Regression Testing Guide
Welcome to this introduction to regression testing. After reading this guide, you will understand how regression testing differs from other types of software testing, why it is important, and common techniques. This guide also introduces strategies to prioritize test cases for regression testing, automate testing, and conduct effective regression testing in agile environments. If you are brand-new to software testing, be sure to also read the Ranorex Beginner’s Guide for User Interface Testing.
Watch our on-demand webinar
Get started in test automation with regression testing: Reduce delivery cycles, expand test coverage, and work more efficiently with this free on-demand webinar.
What is regression testing?
When new software is released, the need to test new functionality is obvious. Equally important, however, is the need to re-run old tests that the application previously passed, to ensure that new software does not re-introduce old defects or create new ones: side effects called regressions. While regression testing is an apparently simple concept, it can be quite challenging in practice due to limited resources such as time, personnel, and test systems.
Example of regression testing: when adding a new payment type to a shopping website, re-run old tests to ensure that the new code hasn’t created new defects or re-introduced old ones.
The discovery of a defect contributes to a team’s knowledge base about the application under test (AUT). That knowledge is captured by creating a test case for the defect and adding the test case to a regression suite. A regression suite contains all of the test cases that will be executed the next time that new software is ready for release. In a large or complex application, it can be helpful to organize the regression suite by application modules. This makes it easier to select a subset of regression test cases when software changes affect only a few modules of the AUT.
What is the difference between regression testing, unit testing, and other types of testing?
Regression testing is a subset of all other types of testing, including desktop testing, web testing, and mobile testing. In theory, regression testing can be done at any stage in the software development lifecycle: including unit, integration, system, and acceptance testing. However, in practice, the term “regression testing” is often used to describe just the “integration testing” process, during which both unit and integration tests are run against the merged AUT.
| Testing Level | Description | Regression Testing Strategy |
Unit testing |
Checks the functionality of individual modules, commonly using mocks or stubs in the place of interface/API calls. | Re-run previously-passed tests for each modified module |
Integration testing |
Combines individual modules and checks their interaction; includes interface/API testing as well as integration testing with back-end services. | Re-run previously-passed unit and integration tests, including all tests for units that changed and priority tests for modules that did not change |
System and acceptance testing |
Verifies that complete, integrated system works as designed and meets the needs of users; includes GUI testing | Re-run system and acceptance tests to verify functionality, as well as performance, accessibility, compatibility, security, etc. |
Why is regression testing important?
Any kind of change can introduce a new defect or allow an old one to recur: new features, enhanced features, bug fixes, and new integrations. Even a single line of code can have potentially serious effects. In 2012, a single extra line that contained the command “goto fail” caused a highly-publicized SSL vulnerability that put the data of millions of people at risk.
Regression testing is an essential element of software quality assurance. During unit and integration testing, regression testing can help catch defects early and thus reduce the cost to resolve them. It creates confidence that an application is ready for deployment. Most importantly, regression testing verifies that code changes do not re-introduce old defects, improving the quality of deployed software.
How to do regression testing
The following is a summary of the regression test process. For more information about the terms and techniques in this summary, refer to the sections below.
Regression testing process
- Developers execute unit-level regression tests to validate code that they have modified, along with any new tests they have written to cover new or changed functionality.
- The changed code is merged and integrated to create a new build of the AUT.
- Smoke tests are executed for assurance that the build is good (“green”) before any additional testing is performed. These tests may be automated and can be executed automatically by continuous integration servers such as Jenkins, Hudson or Bamboo.
- Once there is a “green build,” sanity testing confirms that new functionality works as expected and known defects are resolved before conducting more rigorous integration testing.
- Integration regression test cases are executed to verify the interaction between units of the application with each other and with back-end services such as databases. It is not uncommon for this step to be called “regression testing,” although regression test cases are executed at every stage of development.
- System and user acceptance test cases are executed to verify the application’s functionality, as well as performance, accessibility, compatibility, security, etc. Depending on the size and scope of the released code, either a partial or a full regression may take place. Defects are reported back to the development team and may require additional rounds of regression testing to confirm their resolution. See the How to prioritize test cases section for more information on how to prioritize regression tests.
- Test case maintenance is completed. This includes documenting the number of defects found by each new test case, indicating whether new test cases will be added to the regression set, and identifying test cases to be automated in the next development iteration as well as test cases to be removed from the regression set. Consider archiving rather than deleting test cases removed from the regression set – every test case represents knowledge about the AUT and may have future value.
What are common regression testing techniques?
Unit regression testing
Immediately after coding changes are complete for a single unit, a tester – typically the developer responsible for the code – re-runs all previously-passed unit tests. In test-first development approaches, such as Test-Driven Development (TDD) and in continuous development environments, automated unit tests are built into the code, making unit regression testing very efficient in comparison to other types of testing.
Smoke testing
Smoke testing, also called build verification testing, is a series of high-priority regression tests that are executed after code changes are merged, and before any other testing. The purpose of smoke testing is to quickly ensure that basic functionality is in place before doing additional testing, such as the ability of the AUT to launch and allow users to log on. It is a good practice to conduct smoke testing automatically every time that software changes are pushed to the repository.
Sanity testing
Sanity testing is a subset of functional testing that examines only the changed modules. The goal of sanity testing is assurance that new features work as designed and that defects reported in prior testing have been resolved. Sanity testing has two goals: first, to ensure that the code changes have the desired effect, which may be done through manual, exploratory testing; second, to detect possible regressions, commonly using scripted or automated test cases.
Complete regression
Also known as “retest all” technique, all regression test cases are executed in a complete regression. While a complete regression may be tempting for assurance that the application has been thoroughly tested, this is by definition costly and is not always practical, especially for minor releases. In general, a full regression test suite may be necessary for major releases with significant code changes, following major configuration changes such as a port to a new platform, or to assure compatibility with an updated operating system. A complete regression may also be needed when adapting an application for another language or culture, called “localization.” Localization regression tests verify screen content and user interface settings such as date and time display options, currency and telephone number formats, proper support character sets, correct help and error messages, the layout of GUI elements and dialog boxes, and more. Finally, a complete regression may also be run in cases when “regressions are hard to predict and localize, such as CSS regressions where one change may affect multiple pages and UI states.
Partial regression
As an alternative to a complete regression, a partial regression strategy selects only certain tests to be run. Tests may be selected based on the priority of the test case, or they may be chosen based on the particular module(s) of the AUT that they cover. For example, if a team releases a change to the payment types accepted for an online store, they may choose to conduct regression tests for the payment process, but exclude regression tests for other features such as searching for items and placing them in the cart. See below for suggestions on how to prioritize regression test cases in order to manage limited testing resources.
How to prioritize regression test cases
As an application matures, it inevitably collects a lot of regression test cases. Although test automation tools can significantly decrease the time and effort required for regression testing, it may still be impractical to run the entire suite for every software change. However, running only unit-level regression tests may not be sufficient to expose critical defects. An effective regression strategy can help prioritize regression test cases to ensure that the most critical test cases are executed given the available time and testing resources, and that no important regression tests are missed.
The challenge of regression testing: how to execute enough test cases with the time, staff and system resources available.
The following prioritization strategy assumes four levels of priority: critical, high, medium, and low, and includes strategies developed by Trevor Atkins at www.thinktesting.com.
Test Case Prioritization Strategy
- Begin by selecting test cases that verify critical functionality and core features to identify those that pose the greatest risk of failure. This can be because the feature itself is critical to the system (high level of risk), or because the functionality is complex and prone to failure. A risk assessment matrix can be a useful tool for assigning a risk level. Plan to perform all critical tests as part of smoke testing, prior to any other regression tests.
- Identify test cases that have high code coverage or have found a lot of defects in the past, and assign these a high priority.
- Also, assign high priority to functional test cases that verify the “happy path/happy day scenario” – ones which cover the most common use cases for the AUT from end-to-end. For example, the happy path for a web store would cover all of the steps from user login through item search, check-out, and payment.
- Give all negative test cases a default medium priority. Also known as the “sad path,” negative test cases verify handling of situations such as entry of a bad username or password in a login attempt. The negative test case passes when the application handles the error appropriately, such as displaying an error message to the user and preventing login.
- Likewise, give a default medium priority to all validation tests, including “bad path” tests, which cover boundary/edge type cases as well as corner cases and special values. Boundary/edge cases ensure that the AUT handles the maximum and minimum values that should produce a positive result. A corner case tests a combination of boundary/edge cases.
- Depending on the particular application and the nature and type of code changes, you may decide to treat non-functional tests as low priority for regression testing. However, especially when dealing with customer data that should be kept secure, consider prioritizing security testing as high, or even critical.
- Temporarily remove any test cases from the regression suite that will fail due to existing defects, which have not yet been addressed.
- Review all test cases with medium and low priority, and raise the priority of any that cover areas of special risk, such as security, or test cases that verify compliance with regulations.
- To manage the size of the regression test suite, periodically archive obsolete regression tests that no longer serve a purpose. These could be test cases that cover functionality which no longer exists in the application, or medium- and low-priority tests that have not revealed any defects in several rounds of regression testing.
Regularly review and remove old regression tests to manage the size of the regression test suite.
How to ensure there are enough regression tests
After prioritizing regression test cases, it may be helpful to identify how much of the AUT is covered by the regression test suite. One common approach is to create a code coverage report, and another is to create a requirements traceability matrix.
Code coverage report
A code coverage report is a measurement of how much of a given application’s code is covered by unit tests. While it does not measure how good the tests are, or how good the code is, this report can help to pinpoint areas that may need additional testing. Coverage reports can be generated by automated code coverage tools. Coverage can be measured several ways, such as by lines of code, statements, functions, or use paths. There are many code coverage tools available: the Atlassian website has a helpful comparison of code coverage tools.
Requirements traceability matrix
Regularly review and remove old regression tests to manage the size of the regression test suite.
Wikipedia contributors, “Traceability matrix,” Wikipedia, The Free Encyclopedia, (accessed November 13, 2017).
A requirements traceability matrix shows whether there is adequate test coverage for important functionality. A requirements traceability matrix is often formatted as a spreadsheet that compares the requirements to the test suite, to ensure that there is a test case for every requirement. A traceability matrix is quite flexible: you can create one that just verifies test coverage of new functionality, of both new test cases and regression test cases, or just to ensure adequate coverage by regression test cases.
In the sample traceability matrix above, test cases appear in the left column while the requirements are in the first row. An “x” indicates a test case that covers a given requirement. The “Reqs Tested” column summarizes the number of requirements tested by each test case.
Because full traceability matrices are expensive to create and maintain manually, some agile experts question their benefits and suggest creating one only when required for regulatory compliance or due to project complexity. Costs can be reduced by generating the traceability matrix automatically with a test management tool such as Testrail. This will also make it easier to update the requirements traceability matrix after each development iteration with new test cases that will become regression tests in the next development iteration.
How to plan for system and acceptance regression testing
To create a system-level regression test plan, begin by defining the entry and exit criteria for regression testing, as well as a time boundary for the testing. The time boundary is important, as no code changes or test data changes should be permitted during regression testing.
- Entry criteria may include that all software changes have passed unit testing, a green build has passed smoke and sanity testing, and the regression test suite is prepared.
- Exit criteria may include that all priority test cases have been executed, all defects have been reported, all defects have either been resolved or deferred for future resolution, and the product owner or customer has signed-off.
Identify the test cases that will be part of the regression test suite. Include test cases for recently-changed code and for defects that were recently found and fixed. Exclude test cases for known defects that have not yet been resolved. Finally, automate test cases that are reusable for coverage of core features.
In the plan, be sure to allow time for exploratory testing, as merely re-running tests that the system has already passed may not be effective against hidden defects. Also, allow time to conduct post-testing reporting and analysis, and to maintain regression tests. You should have a version control system for the regression tests themselves, whether they are automated or are text scripts.
Regression testing in agile environments
The short release cycles in agile environments can create challenges for testers. To avoid becoming a bottleneck, test cases must be ready to execute as soon as coding completes on a user story, but this may be difficult if there is a lack of documentation at the beginning of the sprint. The team may feel that it is caught between the proverbial rock and hard place: Delaying testing into a subsequent sprint incurs technical debt that the team will have to pay later, while performing only limited testing increases the risk that defects will make it into production.
Below are some strategies that are helpful for maximizing quality assurance with the limited time available in an agile life cycle:
- Create a task in the sprint to automate the tests selected for regression testing from the previous development iteration. Complete this task early in the sprint.
- Have developers create automated unit tests for new code – which they are already doing if your team follows test-first practices.
- Build time into the sprint for manual, exploratory testing of the stories in the current sprint. Test each story as development finishes – don’t wait until the end of the sprint.
- Apply risk-based prioritization to manage the size of the regression test suite.
- Use a CI server to run unit and integration tests for each build of the application.
- It is not necessary to run the entire regression suite for each build, but do run it at meaningful intervals.
- Finally, make sure that your “definition of done” includes that software changes have passed all regression tests.
Regression testing example
The following is an example of the regression testing process:
- An online store that previously accepted only credit or debit cards wants to accept payment by bank transfer (direct debit).
- Unit tests of the payment setup screen confirm that users are able to enter and save their bank details properly. Meanwhile, unit tests of the payment type screen verify that the new payment type appears, and that old payment types are still available.
- Automated integration regression testing verifies that old payment choices still make correct interface calls to payment APIs. Meanwhile, manual testing verifies that the bank transfer payment type makes the correct interface calls to verify the bank details and perform the proper transfer.
- In system testing, GUI regression testing verifies that user can pay with all payment types, cancel payment, choose a new payment, etc. using test cases based on prioritization. In addition, security testing verifies that the payment transactions are secured properly against man-in-the-middle attacks and other types of threats to online transactions. Performance testing demonstrates that the payment process completes efficiently.
- After testing, the team performs maintenance on the regression test suite. New test cases are documented, prioritized and (optionally) automated for inclusion in the next round of testing. Old regression tests are reviewed, documenting the number of defects found by each, as well as the length of time it took to execute each test and a re-assessment of its priority. The regression test suite is also evaluated to identify candidates for removal.
Strategies to automate regression testing
Regression tests are good candidates for automation. They are executed often and are relatively stable, especially when compared to new test cases. Automating your regression testing suite makes it possible to test overnight and in parallel, and tests can be run automatically for each build of the AUT by a continuous integration server. In short, automating your regression tests enables far greater code coverage at a lower cost than would be possible with strictly manual regression testing.
Following are some strategies to get started with automating your regression tests. First, at the end of each development window, label all new test cases as either re-usable for regression testing or not-reusable. Then, evaluate those that are reusable for automation. Those that are reusable should be automated at the beginning of the next sprint.
Treat automated test cases just like software. Test your test cases to confirm that they function as expected. Automated test cases can fail for a variety of reasons unrelated to the AUT, including missing test data and incorrect test environments. In addition, consider using a version control system such as Git, Subversion, or Microsoft Team Foundation Server to manage changes to the source code of your automated regression tests.
With matchless GUI object recognition, support for codeless or coded creation of testing modules, cross-browser and cross-device testing – Ranorex is your go-to solution for automating regression testing. To get started, watch our free evaluation webinar on Ranorex Studio, contact a member of our sales team, or download a free trial version of Ranorex Studio today.
