Regression Testing Guide
Welcome to this introduction to regression testing. After reading this guide, you will understand how regression testing differs from other types of software testing, why it is important, and common techniques. This guide also introduces strategies to prioritize test cases for regression testing, automate testing, and conduct effective regression testing in agile environments. If you are brand-new to software testing, be sure to also read the Ranorex Beginner’s Guide for User Interface Testing.
What is regression testing?
What is the difference between regression testing, unit testing, and other types of testing?
|Testing Level||Description||Regression Testing Strategy|
|Checks the functionality of individual modules, commonly using mocks or stubs in the place of interface/API calls.||Re-run previously-passed tests for each modified module|
|Combines individual modules and checks their interaction; includes interface/API testing as well as integration testing with back-end services.||Re-run previously-passed unit and integration tests, including all tests for units that changed and priority tests for modules that did not change|
System and acceptance testing
|Verifies that complete, integrated system works as designed and meets the needs of users; includes GUI testing||Re-run system and acceptance tests to verify functionality, as well as performance, accessibility, compatibility, security, etc.|
Why is regression testing important?
Any kind of change can introduce a new defect or allow an old one to recur: new features, enhanced features, bug fixes, and new integrations. Even a single line of code can have potentially serious effects. In 2012, a single extra line that contained the command “goto fail” caused a highly-publicized SSL vulnerability that put the data of millions of people at risk.
Regression testing is an essential element of software quality assurance. During unit and integration testing, regression testing can help catch defects early and thus reduce the cost to resolve them. It creates confidence that an application is ready for deployment. Most importantly, regression testing verifies that code changes do not re-introduce old defects, improving the quality of deployed software.
How to do regression testing
Regression testing process
- Developers execute unit-level regression tests to validate code that they have modified, along with any new tests they have written to cover new or changed functionality.
- The changed code is merged and integrated to create a new build of the AUT.
- Smoke tests are executed for assurance that the build is good (“green”) before any additional testing is performed. These tests may be automated and can be executed automatically by continuous integration servers such as Jenkins, Hudson or Bamboo.
- Once there is a “green build,” sanity testing confirms that new functionality works as expected and known defects are resolved before conducting more rigorous integration testing.
- Integration regression test cases are executed to verify the interaction between units of the application with each other and with back-end services such as databases. It is not uncommon for this step to be called “regression testing,” although regression test cases are executed at every stage of development.
- System and user acceptance test cases are executed to verify the application’s functionality, as well as performance, accessibility, compatibility, security, etc. Depending on the size and scope of the released code, either a partial or a full regression may take place. Defects are reported back to the development team and may require additional rounds of regression testing to confirm their resolution. See the How to prioritize test cases section for more information on how to prioritize regression tests.
- Test case maintenance is completed. This includes documenting the number of defects found by each new test case, indicating whether new test cases will be added to the regression set, and identifying test cases to be automated in the next development iteration as well as test cases to be removed from the regression set. Consider archiving rather than deleting test cases removed from the regression set – every test case represents knowledge about the AUT and may have future value.
What are common regression testing techniques?
Unit regression testing
How to prioritize regression test cases
Test Case Prioritization Strategy
- Begin by selecting test cases that verify critical functionality and core features to identify those that pose the greatest risk of failure. This can be because the feature itself is critical to the system (high level of risk), or because the functionality is complex and prone to failure. A risk assessment matrix can be a useful tool for assigning a risk level. Plan to perform all critical tests as part of smoke testing, prior to any other regression tests.
- Identify test cases that have high code coverage or have found a lot of defects in the past, and assign these a high priority.
- Also, assign high priority to functional test cases that verify the “happy path/happy day scenario” – ones which cover the most common use cases for the AUT from end-to-end. For example, the happy path for a web store would cover all of the steps from user login through item search, check-out, and payment.
- Give all negative test cases a default medium priority. Also known as the “sad path,” negative test cases verify handling of situations such as entry of a bad username or password in a login attempt. The negative test case passes when the application handles the error appropriately, such as displaying an error message to the user and preventing login.
- Likewise, give a default medium priority to all validation tests, including “bad path” tests, which cover boundary/edge type cases as well as corner cases and special values. Boundary/edge cases ensure that the AUT handles the maximum and minimum values that should produce a positive result. A corner case tests a combination of boundary/edge cases.
- Depending on the particular application and the nature and type of code changes, you may decide to treat non-functional tests as low priority for regression testing. However, especially when dealing with customer data that should be kept secure, consider prioritizing security testing as high, or even critical.
- Temporarily remove any test cases from the regression suite that will fail due to existing defects, which have not yet been addressed.
- Review all test cases with medium and low priority, and raise the priority of any that cover areas of special risk, such as security, or test cases that verify compliance with regulations.
- To manage the size of the regression test suite, periodically archive obsolete regression tests that no longer serve a purpose. These could be test cases that cover functionality which no longer exists in the application, or medium- and low-priority tests that have not revealed any defects in several rounds of regression testing.
How to ensure there are enough regression tests
Code coverage report
Requirements traceability matrix
A requirements traceability matrix shows whether there is adequate test coverage for important functionality. A requirements traceability matrix is often formatted as a spreadsheet that compares the requirements to the test suite, to ensure that there is a test case for every requirement. A traceability matrix is quite flexible: you can create one that just verifies test coverage of new functionality, of both new test cases and regression test cases, or just to ensure adequate coverage by regression test cases.
In the sample traceability matrix above, test cases appear in the left column while the requirements are in the first row. An “x” indicates a test case that covers a given requirement. The “Reqs Tested” column summarizes the number of requirements tested by each test case.
Because full traceability matrices are expensive to create and maintain manually, some agile experts question their benefits and suggest creating one only when required for regulatory compliance or due to project complexity. Costs can be reduced by generating the traceability matrix automatically with a test management tool such as Testrail. This will also make it easier to update the requirements traceability matrix after each development iteration with new test cases that will become regression tests in the next development iteration.
How to plan for system and acceptance regression testing
To create a system-level regression test plan, begin by defining the entry and exit criteria for regression testing, as well as a time boundary for the testing. The time boundary is important, as no code changes or test data changes should be permitted during regression testing.
- Entry criteria may include that all software changes have passed unit testing, a green build has passed smoke and sanity testing, and the regression test suite is prepared.
- Exit criteria may include that all priority test cases have been executed, all defects have been reported, all defects have either been resolved or deferred for future resolution, and the product owner or customer has signed-off.
Identify the test cases that will be part of the regression test suite. Include test cases for recently-changed code and for defects that were recently found and fixed. Exclude test cases for known defects that have not yet been resolved. Finally, automate test cases that are reusable for coverage of core features.
In the plan, be sure to allow time for exploratory testing, as merely re-running tests that the system has already passed may not be effective against hidden defects. Also, allow time to conduct post-testing reporting and analysis, and to maintain regression tests. You should have a version control system for the regression tests themselves, whether they are automated or are text scripts.
Regression testing in agile environments
Below are some strategies that are helpful for maximizing quality assurance with the limited time available in an agile life cycle:
- Create a task in the sprint to automate the tests selected for regression testing from the previous development iteration. Complete this task early in the sprint.
- Have developers create automated unit tests for new code – which they are already doing if your team follows test-first practices.
- Build time into the sprint for manual, exploratory testing of the stories in the current sprint. Test each story as development finishes – don’t wait until the end of the sprint.
- Apply risk-based prioritization to manage the size of the regression test suite.
- Use a CI server to run unit and integration tests for each build of the application.
- It is not necessary to run the entire regression suite for each build, but do run it at meaningful intervals.
- Finally, make sure that your “definition of done” includes that software changes have passed all regression tests.
Regression testing example
The following is an example of the regression testing process:
- An online store that previously accepted only credit or debit cards wants to accept payment by bank transfer (direct debit).
- Unit tests of the payment setup screen confirm that users are able to enter and save their bank details properly. Meanwhile, unit tests of the payment type screen verify that the new payment type appears, and that old payment types are still available.
- Automated integration regression testing verifies that old payment choices still make correct interface calls to payment APIs. Meanwhile, manual testing verifies that the bank transfer payment type makes the correct interface calls to verify the bank details and perform the proper transfer.
- In system testing, GUI regression testing verifies that user can pay with all payment types, cancel payment, choose a new payment, etc. using test cases based on prioritization. In addition, security testing verifies that the payment transactions are secured properly against man-in-the-middle attacks and other types of threats to online transactions. Performance testing demonstrates that the payment process completes efficiently.
- After testing, the team performs maintenance on the regression test suite. New test cases are documented, prioritized and (optionally) automated for inclusion in the next round of testing. Old regression tests are reviewed, documenting the number of defects found by each, as well as the length of time it took to execute each test and a re-assessment of its priority. The regression test suite is also evaluated to identify candidates for removal.
Strategies to automate regression testing
Regression tests are good candidates for automation. They are executed often and are relatively stable, especially when compared to new test cases. Automating your regression testing suite makes it possible to test overnight and in parallel, and tests can be run automatically for each build of the AUT by a continuous integration server. In short, automating your regression tests enables far greater code coverage at a lower cost than would be possible with strictly manual regression testing.
Following are some strategies to get started with automating your regression tests. First, at the end of each development window, label all new test cases as either re-usable for regression testing or not-reusable. Then, evaluate those that are reusable for automation. Those that are reusable should be automated at the beginning of the next sprint.
Treat automated test cases just like software. Test your test cases to confirm that they function as expected. Automated test cases can fail for a variety of reasons unrelated to the AUT, including missing test data and incorrect test environments. In addition, consider using a version control system such as Git, Subversion, or Microsoft Team Foundation Server to manage changes to the source code of your automated regression tests.
With matchless GUI object recognition, support for codeless or coded creation of testing modules, cross-browser and cross-device testing – Ranorex is your go-to solution for automating regression testing. To get started, watch our free evaluation webinar on Ranorex Studio, contact a member of our sales team, or download a free trial version of Ranorex Studio today.