Regression Testing Guide
Watch our on-demand webinar
Get started in test automation with regression testing: Reduce delivery cycles, expand test coverage, and work more efficiently with this free on-demand webinar.
What is regression testing?
What is the difference between regression testing, unit testing, and other types of testing?
|Testing Level||Description||Regression Testing Strategy|
|Checks the functionality of individual modules, commonly using mocks or stubs in the place of interface/API calls.||Re-run previously-passed tests for each modified module|
|Combines individual modules and checks their interaction; includes interface/API testing as well as integration testing with back-end services.||Re-run previously-passed unit and integration tests, including all tests for units that changed and priority tests for modules that did not change|
System and acceptance testing
|Verifies that complete, integrated system works as designed and meets the needs of users; includes GUI testing||Re-run system and acceptance tests to verify functionality, as well as performance, accessibility, compatibility, security, etc.|
Why is regression testing important?
Watch our free on-demand webinar
Be Ready for Parallel Testing with Virtual Machines: See how to prepare for testing in parallel, drawing on the experiences of the Ranorex QA team
How to do regression testing
- Developers execute unit-level regression tests to validate code that they have modified, along with any new tests they have written to cover new or changed functionality.
- The changed code is merged and integrated to create a new build of the AUT.
- Smoke tests are executed for assurance that the build is good (“green”) before any additional testing is performed. These tests may be automated and can be executed automatically by continuous integration servers such as Jenkins, Hudson or Bamboo.
- Once there is a “green build,” sanity testing confirms that new functionality works as expected and known defects are resolved before conducting more rigorous integration testing.
- Integration regression test cases are executed to verify the interaction between units of the application with each other and with back-end services such as databases. It is not uncommon for this step to be called “regression testing,” although regression test cases are executed at every stage of development.
- System and user acceptance test cases are executed to verify the application’s functionality, as well as performance, accessibility, compatibility, security, etc. Depending on the size and scope of the released code, either a partial or a full regression may take place. Defects are reported back to the development team and may require additional rounds of regression testing to confirm their resolution. See the How to prioritize test cases section for more information on how to prioritize regression tests.
- Test case maintenance is completed. This includes documenting the number of defects found by each new test case, indicating whether new test cases will be added to the regression set, and identifying test cases to be automated in the next development iteration as well as test cases to be removed from the regression set. Consider archiving rather than deleting test cases removed from the regression set – every test case represents knowledge about the AUT and may have future value.
What are common regression testing techniques?
Unit regression testing
How to prioritize regression test cases
- Begin by selecting test cases that verify critical functionality and core features to identify those that pose the greatest risk of failure. This can be because the feature itself is critical to the system (high level of risk), or because the functionality is complex and prone to failure. A risk assessment matrix can be a useful tool for assigning a risk level. Plan to perform all critical tests as part of smoke testing, prior to any other regression tests.
- Identify test cases that have high code coverage or have found a lot of defects in the past, and assign these a high priority.
- Also, assign high priority to functional test cases that verify the “happy path/happy day scenario” – ones which cover the most common use cases for the AUT from end-to-end. For example, the happy path for a web store would cover all of the steps from user login through item search, check-out, and payment.
- Give all negative test cases a default medium priority. Also known as the “sad path,” negative test cases verify handling of situations such as entry of a bad username or password in a login attempt. The negative test case passes when the application handles the error appropriately, such as displaying an error message to the user and preventing login.
- Likewise, give a default medium priority to all validation tests, including “bad path” tests, which cover boundary/edge type cases as well as corner cases and special values. Boundary/edge cases ensure that the AUT handles the maximum and minimum values that should produce a positive result. A corner case tests a combination of boundary/edge cases.
- Depending on the particular application and the nature and type of code changes, you may decide to treat non-functional tests as low priority for regression testing. However, especially when dealing with customer data that should be kept secure, consider prioritizing security testing as high, or even critical.
- Temporarily remove any test cases from the regression suite that will fail due to existing defects, which have not yet been addressed.
- Review all test cases with medium and low priority, and raise the priority of any that cover areas of special risk, such as security, or test cases that verify compliance with regulations.
- To manage the size of the regression test suite, periodically archive obsolete regression tests that no longer serve a purpose. These could be test cases that cover functionality which no longer exists in the application, or medium- and low-priority tests that have not revealed any defects in several rounds of regression testing.
How to ensure there are enough regression tests
Code coverage report
Requirements traceability matrix
How to plan for system and acceptance regression testing
- Entry criteria may include that all software changes have passed unit testing, a green build has passed smoke and sanity testing, and the regression test suite is prepared.
- Exit criteria may include that all priority test cases have been executed, all defects have been reported, all defects have either been resolved or deferred for future resolution, and the product owner or customer has signed-off.
Regression testing in agile environments
- Create a task in the sprint to automate the tests selected for regression testing from the previous development iteration. Complete this task early in the sprint.
- Have developers create automated unit tests for new code – which they are already doing if your team follows test-first practices.
- Build time into the sprint for manual, exploratory testing of the stories in the current sprint. Test each story as development finishes – don’t wait until the end of the sprint.
- Apply risk-based prioritization to manage the size of the regression test suite.
- Use a CI server to run unit and integration tests for each build of the application.
- It is not necessary to run the entire regression suite for each build, but do run it at meaningful intervals.
- Finally, make sure that your “definition of done” includes that software changes have passed all regression tests.
Regression testing example
- An online store that previously accepted only credit or debit cards wants to accept payment by bank transfer (direct debit).
- Unit tests of the payment setup screen confirm that users are able to enter and save their bank details properly. Meanwhile, unit tests of the payment type screen verify that the new payment type appears, and that old payment types are still available.
- Automated integration regression testing verifies that old payment choices still make correct interface calls to payment APIs. Meanwhile, manual testing verifies that the bank transfer payment type makes the correct interface calls to verify the bank details and perform the proper transfer.
- In system testing, GUI regression testing verifies that user can pay with all payment types, cancel payment, choose a new payment, etc. using test cases based on prioritization. In addition, security testing verifies that the payment transactions are secured properly against man-in-the-middle attacks and other types of threats to online transactions. Performance testing demonstrates that the payment process completes efficiently.
- After testing, the team performs maintenance on the regression test suite. New test cases are documented, prioritized and (optionally) automated for inclusion in the next round of testing. Old regression tests are reviewed, documenting the number of defects found by each, as well as the length of time it took to execute each test and a re-assessment of its priority. The regression test suite is also evaluated to identify candidates for removal.