Search queries should give the same results every time. To that end, I want to create a test environment with the current build, the current version of the database, the correct APIs (if appropriate) to reach the items I wish to search for, and a specific set of known, reliable data—that is, I want to import a known good data set, likely a database tarball, as quickly as possible.
One way to do this is with poetry text, such as “Jabberwocky” by Lewis Carroll. Saving poems as individual files can provide a solid base, and each poem can be further chopped into multiple pages with a simple Ruby script.
There are two aspects to testing that I want to consider. The first is that I test against a known data set that will not change. I can confirm that the values expected will match the number of words by using an oracle of the original text and making a basic inventory of certain words using the grep command.
The second aspect I need to consider is adding new information to the known data set. This introduces another wrinkle: searching to confirm that new information is there and in the correct proportion to the original data.
Common automation tricks
Let’s take my initial example of “Jabberwocky.” I start with a known good data set that has the values I expect to see. In theory, if I search for a single word (“jabber”) and I find it, then search works. Or does it?
There are a variety of ways indexers can search. If I search for “jabber,” I may get “jabber” by itself, but I may also get “Jabberwocky,” or vice versa. Is that the behavior I want to see? If I enclose the search term in quotes, will that limit my search results to that exact term?
Example: Using a known good data set, do a document search for a particular file type (e.g., .txt files). With my known good data set, I should expect to see a link to documents that contain the correct number of text files. By comparing the results I receive with the expected number of documents, these values should match.
What is the expected behavior if I search for multiple words? Do I expect to see any document that contains any of the words mentioned? A test to search for literal values works, but many searches require the ability to cast a wider net.
Examples:
- Does my search tool allow me to do wildcard searches?
- Does my search tool allow me to use regular expressions?
- Can I create compound searches using keywords like or, and, and not?
- Does the search extend to metacharacters, such as tags?
Testing the order and weight of searches
I want to apply criteria to my results to find the most relevant items, making sure I’m looking at the order of results (which can be impacted when filtering by alphabetical listing, date, or creator) and the weight of results (if a certain term used multiple times shows up higher in the search result listings).
Example: Using my known good data set, search for a particular word (“Jabberwocky”) as well as for a fragment of that word (“jabber”). Do I get more search results from the word fragment? Do I also get the larger word to return in my results? Do I have the ability to change the order of my search results if I change the sorting order? If I search for a word that should not appear in the list, do I get a “zero” result? What if a search yields hundreds or thousands of results?
Search can also use dates and times. Does my search take dates into account? If I add anything to the database and reindex, what will that do to the expected results of tests?
Again, the specifics of the search will have a lot to do with the way that searches are indexed, and new information that has been added may not be picked up and be searchable until an indexing job is completed. Therefore, as part of the test design, when I load the original data set or add new data, I run an indexing job and confirm it has completed before continuing with the test.
Considering special characters
Diacritical characters appear in numerous languages, so it’s important to include them in the criteria, too.
Examples:
- Will I see search results that include characters such as à, æ, ē, ñ, or ø?
- What if I switch the language option for my application? Would I get different results for US English versus UK English?
Things that are hard to automate
I have seen a number of areas that have produced unusual results and required changes in the testing approach. The methods used to filter results need to be tested as much as (if not more than) the actual results I expect to receive.
Does the application present results in the correct date order? What happens if the order is reversed?
Many applications allow for a way to refine or order search results, such as Created By, Modified By, Date Created, Date Modified, and Relevance (usually determined by the number of occurrences of the search item).
Am I including metadata as part of my search? If so, how does it differentiate?
Metadata could be using hashtags or other labeling methods to group like content together. Using the “jabber” example again, it could be a literal search for the word or fragment, but it could also be a tag used to group all of Lewis Carroll’s works together. Does my search take that into account?
What if results span over multiple pages?
Searching for values that appear on a single page or within a set number of tabs displayed allows the ability to point to a specific value. If I need to search over multiple pages, I likewise have to take into account exactly which page I’d expect a particular result to appear on and confirm that I can find it. If I make multiple changes to the data set, the ability to locate those values becomes more challenging. Implementing an endless scrolling function can help with this, and writing the test to use API calls that can control the number of values returned can simplify this process.
Common themes
In this series, there are going to be some familiar patterns and areas that need to be considered for effective testing. Several of these are areas that could be combined into modules, considering I will likely use them frequently.
How do my role and permissions determine what I should and should not see?
I need to consider not just if it is possible to locate an item in a search, but if I should be allowed to see that item in the first place. Creating roles for users and then removing permission levels can ensure that the searches work as expected.
How do I decide what to test?
There will be tradeoffs to how much can be tested and at what level of detail. The more granularity I am able to use, the more test cases I can create, but of course, the longer it will take to test those options. My suggestion is to start with the simpler cases first, run those and confirm that they are working, and then create additional cases that go into greater granularity. Cover the areas that would expose the most risk first (known data set tests, simple searches, wildcard searches, and roles and permissions), then as time permits, expand the tests so that coverage becomes more complete.
Download your risk-free, full-featured version of Ranorex Studio now to experience the power of test automation.
About the Author
Michael Larsen has, for the better part of his 20+ year career, found himself in the role of being the “Army of One” or “The Lone Tester” more times than not. He has worked for with a broad array of technologies and industries including virtual machine software, capacitance touch devices, video game development and distributed database and web applications. Michael currently works with Socialtext in Palo Alto, CA.
