Continuous testing has become more and more popular in recent years. Teams strive to have automated tests at every stage of the software development lifecycle in order to evaluate risks and obtain immediate feedback. But when the aim is to test continuously, there is one question that teams often struggle to answer: When do we stop testing?
We can validate whether a product is working as expected, but testing has to stop at some point so the product can be released to the customer. Complete exhaustive testing is impossible.
Here are six questions whose answers can help you make the decision that the time is right to stop testing.
1. What risks have been mitigated?
There are risks associated with every feature developed. A good approach to decide when to stop testing is to analyze whether the team has mitigated all the identified risks.
This could have different meanings based on the context of the project, such as:
- Have the tests related to the identified risks been executed? What were the results?
- Are there missing risks that need to be addressed before releasing the product to the customer?
- Does another round of test case execution need to happen to retest the fixed defects?
- How confident are you in releasing the product in its current state?
Risk-based testing helps evaluate the product in the customer’s lens, and mitigating all the risks identified will ultimately define when testing is complete.
2. Are there open critical defects?
Realistically, there are always going to be defects in the product, even after releasing it to production. The only thing we can do is identify and fix defects that would significantly impact the customer. One way to do this is to ensure all the identified critical or high defects are fixed and retested.
3. Are you meeting project deadlines?
There are often strict release schedules to ensure features are released on time. This could be due to multiple reasons, such as signed contracts, getting a competitive edge in the market, or helping retain customers by providing value.
As a result, teams are on the hook to deliver features by certain dates. There should be weeks of planning meetings, multiple release schedules, and clear criteria for deliverables for a certain time period. This helps get more clarity on stakeholders’ goals and expectations.
4. Do you have acceptable requirements coverage?
Before the start of feature development, there is a list of requirements documented in user stories that the team has to work on. One way to know whether testing is complete is to ensure that all the requirements identified for a given release cycle have been tested. Usually, a release cycle is split into different sprints to make this effort more manageable and measurable.
If there are user stories moved to the backlog, the stakeholders have to make an informed decision as to whether those user stories are important for the current release or could be scheduled to go out at a later time.
5. Is the product good to release?
When the project deadline comes around, the stakeholders have to collectively decide whether a product is at an acceptable level to be released to their customers. The factors that aid in this decision-making process could vary based on the project’s context, the planned features to be released, signed contracts, and more.
6. Has the difficulty exceeded the value?
Sometimes it is glaringly clear that the product has reached a certain level of stability or maturity. A great indication for this is when teams are:
- Finding fewer defects with less severity over a certain time period
- Spending more time discussing than testing
- Starting to repeat the same testing steps multiple times and ending up with the same expected results
- Sharing the same status updates in multiple standup meetings
When this is the case, it is common sense to stop testing and move the focus to other modules with higher risks.
Testing teams should have various metrics to measure whether it is time to stop testing and move on to other areas. It all boils down to what the team’s priorities are and how to make your customers happy.
All-in-one Test Automation
Cross-Technology | Cross-Device | Cross-Platform
