This article describes how to use the Ranorex Studio IDE and the Ranorex API for test automation in your behavior-driven development (BDD) process. BDD requires a cognitive shift: instead of thinking about testing functions, you are now thinking about behaviors....
Sample Test Cases – Credit Card Transaction
Learn how an optimized set of scenarios for the credit card processing application is efficiently generated in DesignWise.
This plan is based on the example in Elfriede Dustin’s article “Orthogonally Speaking,” available here.
What are our testing objectives?
We know that testing each item in our system once is not sufficient; we know that interactions between the different things in our system (such as a particular browser type interacting with input methods, for example) could well cause problems. Similarly, we know that the written requirements document will be incomplete and will not identify all of those potentially troublesome interactions for us. As thoughtful test designers, we want to be smart and systematic about testing for potential problems caused by interactions without going off the deep end and trying to test every possible combination.
DesignWise makes it quick and simple for us to select an appropriate set of tests whatever time pressure might exist on the project or whatever testing thoroughness requirements we might have. DesignWise-generated tests automatically maximize variation, maximize testing thoroughness, and minimize wasteful repetition.
What interesting DesignWise features are highlighted in this sample plan description?
Value Expansions – How can I maximize variation without increasing the number of my tests?
Forced Interactions – How to force certain high priority scenarios to appear in your set of tests.
Auto-Scripting – How to save time by generating detailed test scripts in the precise format you require (semi)-automatically.
Coverage Graphs – How to get fact-based insights into “how much testing is enough?”.
Matrix Charts – How to tell which exact coverage gaps would exist in our testing gif we were to stop executing tests at any point in time before the final DesignWise-generated test.
What interesting test design considerations are raised in this particular sample plan?
- First Strategy – General Descriptive Terms (used for Credit Card Number) – Simply describe the nature of the input with the Values of Correct Invalid Length and Invalid Digits
- Second Strategy – Value Ranges: For Quantity Purchased, one value is 5 – 10. When using this value, DesignWise will choose a number between 5 and 10, but first chooses the upper and lower limits (providing boundary testing) and will then at random choose other values from the range. DesignWise will even let you specify decimal value length.
- Third Strategy – Descriptive Terms with Pre-Defined Values (used for Invalid Length, Invalid Digits, and also in This Month or Next Month) – Use Value Expansions to assign specific numbers for each range.
- “General Descriptive Terms” will result in tester instructions like: enter a high income… Potential disadvantage: is this enough guidance for testers who might execute the tests later?
- “Value Ranges” works well when you have equivalence classes defined by numerical ranges. If there are different rules for small loans than large loans, and small loans are defined as loans of anywhere between 10,000 and 99,999, you can enter 10,000 – 99,999. The DesignWise algorithm is smart enough to generate tests for you that automatically cover the valid boundary values in your ranges (e.g., highest and lowest values as well as values randomly selected from inside the range).
- “Descriptive Terms with Predefined Value Expansions” would provide the execution team with more precise instructions than the 1st strategy (e.g., instead of Region 1, instructions would be more specific, such as Big City in Region 1 or Small Town in Region 1).
It is also worth highlighting that no matter whether you want to execute a set of a few dozen, a few hundred, or a few thousand tests, DesignWise can generate a prioritized / optimized set of scenarios for your thoroughness and timing needs.
It is often useful to start by identifying a verb and a noun for the scope of our tests
Designing powerful software tests requires that people to think carefully about potential inputs into the system being tested and how they might impact the behavior of the system. We strongly encourage test designers to start with a verb an a noun to frame a sensible scope for a set of tests and then ask the “newspaper reporter” questions of who?, what? when? where? why? how? and how many?
How or How long/many
- How many items can someone buy?
- How far in the future do Credit Cards typically expire?
- When is (what time of day) the purchase made?
- Why would a credit card be declined? (e.g., invalid length, invalid digits)
- What kind/type of credit cards are accepted?
- What would stop a purchase from being accepted? (e.g., invalid card numbers, invalid expiration dates, invalid quantities, etc.)
- What sort of items can be purchased?
Variation Ideas entered into DesignWise’s Parameters screen
Once we have decided which test conditions are important enough to include in this model (and excluded things – like ‘What time of day is the purchase made?’ – that will not impact how the system being tested operates), DesignWise makes it quick and easy to systematically create powerful tests that will allow us to maximize our test execution efficiency.
Once we enter our parameters into DesignWise, we simply click on the “Scenarios” link in the left navigation pane.
DesignWise helps us identify a set of high priority scenarios within seconds
DesignWise gives test designers control over how thorough they want their testing coverage to be. As in this case, DesignWise allows testers to quickly generate dozens, hundreds, or thousands of tests using DesignWise’s “coverage dial.” If you have very little time for test execution, you would find those 23 pairwise tests to be dramatically more thorough than a similar number of tests you might select by hand. If you had a lot more time for testing, you could quickly generate a set of even more thorough 3-way tests (as shown in the screen shot immediately below).
Selecting “3-way interactions” generates a longer set of tests which cover every single possible “triplet” of Values
The only defects that could sneak by this set of tests would be these two kinds:
- 1st type – Defects that were triggered by things not included in your test inputs at all (e.g., if special business rules should be applied to an applicant living in Syria, that business rule would not be tested because that test input was never included in the test model at all). This risk is always present every time you design software tests, whether or not you use DesignWise.
This risk is, in our experience much larger than the second type of risk:
- 2nd type – Extraordinarily unusual defects that would be triggered if and only if 4 or more specific test conditions all appeared together in the same scenario. E.g., if the only way a defect occurred was if a transaction was for (i) Discover card and a (ii) Correct credit card numbers, with an expiration of (iii) This Month, and they purchased a (iv) book. It is extremely rare for defects to require 4 or more specific test inputs to appear together. Many testers test software for years without seeing such a defect. More details are available here.
If a tester spent a few days trying to select tests by hand that achieved 100% coverage of every single possible triplet of Values (such as, e.g., (i) Amex card, and (ii) invalid credit card number length, and (iii) purchasing a Book, Software, and Video), the following results would probably occur:
- It would take far longer for a tester to attempt to select a similarly thorough set of tests and the tester would accidentally leave many, many coverage gaps.
- The tester trying to select tests by hand to match this extremely high all triples thoroughness level would create far more than 95 tests (which is the optimized solution, shown above).
- Almost certainly, if the tester tried to achieve this coverage goal in 200 or fewer tests, there would be many, many gaps in coverage (e.g., 3-way combinations of Values that the tester accidentally forgot to include).
- Finally, unlike the DesignWise-generated tests which systematically minimize wasteful repetition, many of the tester’s hand-selected scenarios would probably be highly repetitive from one test to the next; that wasteful repetition would result in lots of wasted effort in the test execution phase.
We can force specific scenarios to be included
You’ll notice from the screen shots of 2-way tests and 3-way tests shown above that some of the Values in both sets of tests are bolded. Those bolded Values are the Values we “forced” DesignWise to include by using this feature.
Auto-scripting allows you to turn scenario tables (from the “Scenarios” screen) into detailed test scripts
We document a single test script in detail from the beginning to end. As we do so, we indicate where our variables (such as, “Type of Credit Card,” and “Credit Card Number,” and “Credit Card Expiration Date”) are in each sentence. That’s it. As soon as we document a single test in this way, we’re ready to export every one of our tests.
From there, DesignWise automatically modifies the single template test script we create and inserts the appropriate Values into every test in your plan (whether our plan has 10 tests or 1,000).
We can even add simple Expected Results to your detailed test scripts
It is possible to create simple rules using the drop down menu that will determine when a given Expected Result should appear. To do so, we would use the drop down menus in this feature to create simple rules such as “When ____ is ___ and when ____ is not ____, then the Expected Result would be_____.”
This Expected Results feature makes it easy to maintain test sets over time because rules-based Expected Results will automatically update and adjust as test sets get changed over time.
Coverage charts allow teams to make fact-based decisions about “how much testing is enough?”
This graph, and the additional charts shown below, provide teams with insights about “how much testing is enough?” And they clearly show that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of test sets is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. As explained here, this type of “diminishing marginal return” is very often the case with scientifically optimized test sets such as these.
DesignWise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in DesignWise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.
We know we would achieve ~81.0% coverage of the pairs in the system if we stopped testing after test number 15, but which specific coverage gaps would exist at that point? See the matrix chart below for that information.
The matrix coverage chart tells us exactly which coverage gaps would exist if we stopped executing test before the end of the test set
For example, in the first 15 tests, there is no scenario that includes both (a) “Type of Credit Card – Discover” together with (b) “Credit Card Number – Correct.”
We can analyze coverage on the extremely thorough set of 3-way tests we created also.
Mind maps can be exported from this DesignWise plan to facilitate stakeholder discussions.
Detailed test scripts (complete with stepped-out tester instructions and rule-generated Expected Results) can be exported also:
Other possible export formats could include test data tables in either CSV or Excel format or even Gherkin-style formatting.