This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors....
Defect Analysis After DesignWise
How to identify the defect reason after you move to combinatorial testing.
A common question from DesignWise clients in early adoption stages is:
“Before, we had a clear testing focus for each scenario, so we knew exactly what would fail the test. Now, with all the inputs mixed together, how can we identify the cause of a defect?”
The purpose of this article is to demonstrate a defect analysis approach when using DesignWise tests. We’ll use a simplistic customer preferences application with 5 “Checked”/ ”Not Checked” indicators, looking like this:
If instead we use DesignWise to generate the a set of 2-way tests for this application, we might get test scenarios like these:
An Analysis Approach
First, let’s define the “strength” of the defect. We’ll call an issue a “one-way defect” if it is caused by just a single parameter value, irrespective of everything else happening in the test case. A “two-way defect” means that a specific combination of 2 parameter values happening together triggers the error (e.g., Checked for Indicator 1 AND Checked for Indicator 4), and so on for “three-way defects” and beyond.
Keeping that in mind, let’s go back to the original manual selected scenario #3:
Execute all the scenarios and notice that scenarios #1, #4, and #6 were the only ones that failed.
Analyze what is the consistent element in each of these scenarios:
- Indicator 1 has both values – not a culprit
- Indicator 2 has both values – not a culprit
- Indicator 3 has only “Checked” value – culprit
- Indicator 4 has both values – not a culprit
- Indicator 5 has both values – not a culprit
As in this case, the holistic view of the DesignWise scenarios usually provides you with enough data to pinpoint an n-way issue.
At the same time, the original manually selected scenarios didn’t cover two indicators checked simultaneously, which means there’s a significant risk of 2-way defects (or higher) slipping into production. Based on the statistical evidence, on average, 84% of defects in production are caused by 1 or 2 system elements acting together to create a defect.
So it’s important to use a combinatorial testing methodology to improve the testing coverage and then adjust the defect analysis process accordingly.
An Analysis Approach
Alternatively, you can use a higher coverage strength, especially if the system has no more than 6 binary parameters. In our demo example, selecting “5-way” from the drop-down automatically provides us with all possible combinations of 5 inputs.