Testing is a crucial part of the software creation process. It ensures that your code is working correctly and that all bugs are found before the software lands in the hands of consumers. But it can be difficult to explain these tests and their results to investors...
Defect Analysis After DesignWise
How to identify the defect reason after you move to combinatorial testing.
A common question from DesignWise clients in early adoption stages is:
“Before, we had a clear testing focus for each scenario, so we knew exactly what would fail the test. Now, with all the inputs mixed together, how can we identify the cause of a defect?”
The purpose of this article is to demonstrate a defect analysis approach when using DesignWise tests. We’ll use a simplistic customer preferences application with 5 “Checked”/ ”Not Checked” indicators, looking like this:
If instead we use DesignWise to generate the a set of 2-way tests for this application, we might get test scenarios like these:
An Analysis Approach
First, let’s define the “strength” of the defect. We’ll call an issue a “one-way defect” if it is caused by just a single parameter value, irrespective of everything else happening in the test case. A “two-way defect” means that a specific combination of 2 parameter values happening together triggers the error (e.g., Checked for Indicator 1 AND Checked for Indicator 4), and so on for “three-way defects” and beyond.
Keeping that in mind, let’s go back to the original manual selected scenario #3:
Execute all the scenarios and notice that scenarios #1, #4, and #6 were the only ones that failed.
Analyze what is the consistent element in each of these scenarios:
- Indicator 1 has both values – not a culprit
- Indicator 2 has both values – not a culprit
- Indicator 3 has only “Checked” value – culprit
- Indicator 4 has both values – not a culprit
- Indicator 5 has both values – not a culprit
As in this case, the holistic view of the DesignWise scenarios usually provides you with enough data to pinpoint an n-way issue.
At the same time, the original manually selected scenarios didn’t cover two indicators checked simultaneously, which means there’s a significant risk of 2-way defects (or higher) slipping into production. Based on the statistical evidence, on average, 84% of defects in production are caused by 1 or 2 system elements acting together to create a defect.
So it’s important to use a combinatorial testing methodology to improve the testing coverage and then adjust the defect analysis process accordingly.
An Analysis Approach
Alternatively, you can use a higher coverage strength, especially if the system has no more than 6 binary parameters. In our demo example, selecting “5-way” from the drop-down automatically provides us with all possible combinations of 5 inputs.