Nowadays, web visitors don't only consider the usefulness of a website but base their decision to stay or leave the site on the layout, design, and functionality. A significant portion of visitors abandon a website with poor functionality or confusing navigation,...
How can I best identify variation in the system I’m testing?
Learn how to identify & evaluate Parameter candidates in your systems under test and how to optimally include them in your DesignWise plans.
At the risk of stating the obvious, you’ll need to use your judgment here; not all of these categories will be necessary (or even useful!) to include in all of your sets of tests.
An overview of variation ideas
Manufacturer: Dell, Lenovo, Apple
Processor: Intel, AMD
Age: manufactured within last 2 years, manufactured 2 or more years ago
Software configuration inputs could include:
Operating System: Windows 7, Windows 8, Windows 10, Mac OS
Browser: IE10, IE11, Firefox, Chrome, Safari
Cookies: cookies enabled, cookies disabled
Channel inputs could include:
Channel: typical web transaction, typical web transaction, phone call center (Note: double-weighting ‘typical web transaction’ will make it show up twice as often as ‘phone call center’)
Additional Environmental Variations involve file types and storage locations
Flight schedule information for flights is stored: Database Huey, Database Dooey, Database Louie
Frequent Flier Mileage data stored in: United database, (Frontier Airlines database; not yet transferred into post-merger database), N/A
For example, User Variations that may be considered in end-to-end tests of hipmunk.com, might include:
User type: normal user (without prior account), normal user (with established account), call center agent
User persona: power user, clueless newbie user, typical user
Account characteristics: Gold Frequent Flier, Silver Frequent Flier, Bronze Frequent Flier, not a Frequent Flier
Other related ideas:
User navigates primarily using: keyboard (hot-keys wherever possible), keyboard (no hot-keys), mouse
User rapidly clicks multiple times on submit buttons and next buttons: no, yes, no (Note: double-weighting ‘no’ will make it appear twice as often as ‘yes’)
User enters information from top of screen to bottom of screen: always, usually, never
User enters all required information the first time they submit information: always, usually, never
Actions that could vary from test to test in our hipmunk.com example could include:
- One Way or Round Trip: One Way, Round Trip
- Destination: South America, Europe, Asia, Africa, Australia, North America
- Class of Travel: Economy, Business, First
- Special Meal Requested: None, None, Vegetarian, Kosher (Note: double-weighting ‘None’ will make it show up twice as often as ‘Vegetarian’ or ‘Kosher’)
- Saturday Night Stay-over: Yes, No
Data that might vary will often be inextricably interrelated to usage examples (directly above). Each of the user selections above, for example, would create data that should be stored by the system. That’s one of the reasons we categorize “Actions” and “Data” together as “Usage Variations.” Even so, “Data” can be useful as a trigger for additional testing ideas such as quantities, data ranges, and specific data formats. Here are some examples of test inputs that could be included from this example:
- Date format: mm/dd/yy, dd/mm/yy, mm/dd/yyy, dd/mm/yyyy
- Number of passengers: 1, 2-4, more than 4
- Special characters included in customer name?: no, yes, no (Note: recall double-weighting)
“Hendrickson Variables” offer a great checklist to tick through as you think about possible variables to add.
Hendrickson lays-out her list of Interesting Things to Vary as follows:
Countable Things –
Relative Positioning –
Files and Storage-
Geographic Locations –
Timing, Frequency, and Duration –
Input and Navigation –
A “rule of thumb” to guide you as you decide what things should be included and excluded in the Inputs screen:
INCLUDE – If you have reason to believe that including a test input is likely to matter and it would not take much extra time to vary that idea in each of the tests in your test set, then include it. An example: if you’re testing a transactional web application and prior releases have had quite a few defects associated with the IE10 browser, include “Browser Type” as a Parameter and “IE10” as one of your Values.
EXCLUDE – Conversely, if you suspect that a particular test idea is not going to make any difference and that variation would take considerable time to include in each of your tests, then do not enter those into the “Inputs” screen of DesignWise. An example: as pointed out by James Bach, defects have been triggered by blocked cooling vents above servers on extremely rare occasions . This caused a server to overheat. Even so, it would rarely be sensible to invest the time necessary to test out multiple test scenarios with overheated servers on a given set of tests.
Including test ideas using the Value Expansion feature allows for quick testing without wasting time (over testing).
An example: imagine that you’re testing a new feature of a financial transaction application. It will be used for foreign or domestic transactions. Setting up the testing environment for each foreign transaction requires dramatically more time than for domestic transactions. If you were to include “Type of Transaction” as a Parameter and use “Domestic” and “International” as the 2 Values for that Parameter, approximately 50% of your tests would be international transactions which would each require a great deal of extra setup time.
In such situations, you might instead consider using “Type of Transaction” as a Value and then adding “Domestic, International, Domestic, Domestic, Domestic, Domestic, Domestic” as sub-values when you create your Value Expansion. Using this test design technique would result in at least one “International” transaction (for coverage of this potentially important scenario) but you would avoid spending too much time “over-testing” the idea.
When you’re first getting started, feel free to experiment!
Your ability to make judgment calls will increase as you gain experience. When you’re experimenting, don’t be afraid to add a few more test ideas into the scope of your test sets than you’re used to. You will notice that when you design tests with DesignWise, you can include more test ideas in the scope of your tests than you’re used to including. That’s because:
- You will often find surprising defects when you add variation to your tests, yet adding the variation to your tests often hardly takes any time at all.
- Adding a Parameter with a few Values takes only a few seconds on the “Inputs” screen of DesignWise.
- Finally, unlike when you select and document tests by hand, generating the tests with the newly-added test ideas won’t add noticeably more test documentation time; the tests generated by DesignWise will all automatically include that test idea.
- Provided you keep the number of Values per Parameter small, you usually won’t increase the number of tests that DesignWise will generate.