Less Brittle GUI Test Automation, Part 1/2

“Once test automation runs a single time, and the programmers make a change, it can cease to be test automation and become change detection.” – Matthew Heusser

Make a change to the software under test and re-run the GUI test automation. The result might be a large set of errors, which then need to be reproduced and marked as bugs; or the tests need to be updated. This is an issue that can be unsettling to those new to test automation. It’s especially common with older record/playback tools that might require the author to re-record the entire test. Make no mistake, the record and playback approach is very effective for a quick start to automation projects. But over time, tests tend to become “brittle” if we don’t take into account the changing nature of software and how we interact with it.

Here are a few ways tests become brittle at the GUI level—and how to make them more robust.

Reliance on Labels

Let’s take one of the most common automation tasks, log in. If I were to survey a number of login interfaces, I would find that the labels — the text immediately before the first text field — can vary considerably. Entries can be listed as Username, User Name, User ID, Account Name, Account Number, Mobile ID, etc. These are then typically followed by a text box. Using the label for the value to interact with can be problematic since someone might say “hmmm, I think I’d rather have our login page say “User ID” rather than “Username”. If I use the label to identify the element, that will likely break my test. A few more label changes and I have a lot of broken tests. The Password field is more standard, but even that has variations as well. Even if the words themselves are kept consistent, one of the most likely ways to break this test is to change the language or locale. Username and Password may be consistently used and they may never be changed, but if the locale changes, especially to a locale that is not English, those labels will not be there and the test will fail.

So how do I resolve this issue? I use a value that I know will not change or is very likely to not change. Rather than look for the literal word “Username”, I look for an element ID, which might be labeled something like “login_username”. instead of looking for the Password label, I look for the ID called “login_password”. The button to complete the task should likewise have an ID, like “submit_login_info”. Why should I use these values instead? Because these IDs are more likely to remain consistent than the labels are, especially if we take into account a change of locale. By using the IDs as my identifier, my tests should pass regardless of the language or locale selected.

Without an ID, some automation tools query the interface to find a connection, such as “first text box on the screen.” If programmers add a search box to the top-right, the test will fail.

Ask your programmers for named ID’s: that is a feature for testability.

Using Specific Data for Tests

When starting to write a test for the log in function, it makes sense to have a single user, a single password, and some data that will be easy to verify the functionality for that test. That’s all well and good. But what happens if I want to make this test work with many different users? Perhaps I want to test a variety of scenarios and pages to be displayed. Perhaps I want to confirm that a particular item is displayed on a user’s page, but the value of that element may be different for each user. Here’s a simple scenario. Let’s use the login example again. If I log in as “mlarsen” and a password of “P@ssw0rd”, then click submit, I can look to see that my full username of “Michael Larsen” displays in the upper right‑hand corner. That’s an OK start, but it’s going to be limiting. I know it will appear correctly for me, but how about for someone else? How about for “Kärl Køstling”? How about for “ちはる しおだ” (Chiharu Shioda)?

So how do I resolve this issue? I make the test use variable names instead of literal text strings and I create an external file, database, or table that stores the variables needed for each test. Again, I like to use characters that are less common to see if they will display correctly. To this end, I create a text file with the following example data:

username,password,displayname
michael.larsen,P@ssw0rd,Michael Larsen
karl.kostling,P@ssw0rd,Kärl Køstling
chiharu.shioda,P@ssw0rd, ちはる しおだ

From there, I set up my tests to use variables named ${username}, ${password}, and ${displayname} to allow me to use multiple values and confirm that what is displayed accurately reflects the values stored. An additional benefit is that these tests can be expanded to as many values as I want to test with. Two or ten thousand, it shouldn’t make a difference.

For that matter, if I expect different results, I can put those in the table as well.

XPATH to Find Specific Objects

Sometimes there is no alternative but to use XPATH to find elements on a web page or mobile application. However, it’s also important to not make it too difficult or too specific for the user writing the test to have to look for items. If I have to look for the fourth instance of a button with a label of “button” to accomplish a workflow task, that already tells me we’re in problem territory. However, there may be a very good reason for this, so what do I do when I don’t have an option to ask for a unique ID? In that case, I can use a few tricks that will get me to what I want. One of the ways to do this is to consider where in the page the element I want to interact with is located. By doing that, I can narrow down how far I want/need to look for an element. True, I may have no control over the fact that I have multiple buttons, but if I know where the particular button I want to work with is located, then I can define a locator that anchors that element, or at least better than it would be if I tried to pick that specific button. For example, the XPath below looks for the second paragraph under the first div under an element that does have an ID, called “post”:

//*[@id="post-7"]/div/p[2]

That is preferable to an extremely long XPATH that covers the entire page, with seven levels of divs that might change.

Note: to uniquely identify UI elements in a way that is robust and reliable even when the user interface changes, Ranorex uses a subset of the XPath syntax with extensions, called the RanoreXPath.

Lessons for Today

For today, consider named IDs, data-driven tests that can repeat steps but expect different results, and the use of as precise XPATH as possible.

Next time, we’ll talk about comparing images, explicit waits, and driving input from places other than the user interface.