Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
Read our comprehensive guide to learn what GUI testing is, how to create an effective GUI test plan, and how to build GUI test cases.
What Is GUI Testing?
The term GUI stands for “graphical user interface,” meaning the part of a software application that is visible to the user. A GUI may contain menus, buttons, text boxes, images, and other elements. One of the first successful GUIs was the Apple Macintosh, which popularized the concept of a user “desktop” complete with a calendar, trash can, and calculator.
An early GUI: The Apple Macintosh released in 1984.
Image source: folklore.org CC license
In the case of a calculator application, GUI testing would include verifying that the application responds correctly to events such as clicking on the number and function buttons. It would also confirm that appearance elements such as fonts and images conform to design specifications.
But in today’s testing environment, GUI is no longer limited to the desktop of a computer. It may be a mobile app that is available on all the major mobile platforms, or it may be a cloud application that must be supported by all the major browsers. Testers must perform cross-browser and cross-platform testing to identify defects and ensure that the application fulfills all requirements.
Graphical user interface testing is performed from the user’s perspective, instead of a developer’s. It occurs during system testing, which is a black-box testing method where no knowledge of the underlying code is necessary. System testing verifies that a complete, integrated system works as designed.
UI vs. GUI Software Testing: What’s the Difference?
User interface (UI) testing is similar to GUI testing, and the two terms are often treated as synonyms. However, UI is a broader concept that can include both GUI and Command Line Interfaces (CLIs). A CLI allows a user to interact with a computer system through text commands and responses. Although CLIs predate GUIs, they are still in use today and are often preferred by system administrators and developers.
Tip
To get to the command line on a Windows PC, start the Command Prompt desktop application. On a Mac, open the Terminal application
While there is no universally accepted definition of testing terms, a good source is the International Software Testing Qualifications Board (ISTQB) in their Certified Tester Foundation Level Syllabus.
What Are the Popular GUI Testing Techniques?
Testing techniques describe how to test a target application, also known as the “application under test” or AUT. This section looks at three major GUI testing techniques: scripted testing, exploratory testing, and user experience testing.
The V-model of software development identifies testing tasks for each stage of development.
Scripted Testing
In scripted testing, software testers design and then execute pre-planned scripts to uncover defects and verify that an application does what it is supposed to do. For example, a script might direct a tester through the process of placing a specific order on an online shopping site. The script defines the entries that the tester makes on each screen and the expected outcome of each entry. The tester analyzes the results and reports any defects that are found to the development team. Scripted testing may be performed manually or supported by test automation.
Scripted Testing Benefits
Because scripted testing is pre-planned and has tangible outputs (the test scripts and testing reports), scripted testing gives product managers and customers confidence that an application has been rigorously tested. By creating test scripts early in the development process, teams can uncover missing requirements or design defects before they make it into the code. While test scripts must be created by an experienced tester with knowledge of the system, less experienced testers can perform the actual testing. Finally, test scripts can be reused in the future for regression testing, and can also be automated for greater efficiency.
Scripted Testing Challenges
Scripted testing requires a lot of up-front planning, which can cause time pressure, especially in agile development environments. Test scripts must be updated as the AUT changes. More importantly, studies have suggested that the rigid structure of test scripts may cause testers to miss defects that would be uncovered by exploratory testing. They also suggest that scripted GUI tests require too long to execute manually for the results they produce.
Exploratory Testing
Rather than following pre-written test scripts, exploratory testers draw on their knowledge and experience to learn about the AUT, design tests, and then immediately execute them. After analyzing the results, testers may identify additional tests to be performed and/or provide feedback to developers.
Although this GUI testing method does not use detailed test scripts, it still uses pre-planning. For example, in session-based exploratory testing, testers create a document called a test charter to set goals for the planned tests and set a time frame for a period of focused exploratory testing. Sessions of exploratory testing are documented by a session report and reviewed in a follow-up debriefing meeting.
Likewise, during scripted testing, there may be some decisions available to testers including the ability to create new tests on the fly. For that reason, it’s helpful to think of scripted testing and exploratory testing as being two ends of a continuum rather than polar opposites.
Both scripted and exploratory testing can be completely manual, or they can be assisted by automation. For example, an exploratory tester might decide to use test automation to conduct a series of tests over a range of data values.
Exploratory Testing Benefits
As time planning and writing test cases is reduced, testers have more time to focus on the actual testing of the AUT. Testers who are challenged to use their knowledge, skills, and creativity to identify defects and ensure conformance to requirements may be more engaged and may find more defects than testers who are restricted to scripts written by others.
Exploratory Testing Challenges
This GUI testing method requires testers to have a deep understanding of the performance requirements of the AUT as well as skills in software testing. Due to the realities of time constraints and resource availability, it may be impractical to try to cover an entire AUT with exploratory testing. These tests are not as repeatable as scripted tests, which is a major drawback for regression testing. Further, relying on exploratory testing alone can create concern in product managers or customers that code will not be covered and defects will be missed.
User Experience Testing
In user experience testing, actual end-users or user representatives evaluate an application for its ease of use, visual appeal, and ability to meet their needs. The results of GUI testing may be gathered by real-time observations of users as they explore the application on-site. Increasingly, this type of testing is done virtually using a cloud-based platform. As an alternative, project teams can do beta testing, where a complete or nearly complete application is made available for ad hoc testing by end users at their location, with responses gathered by feedback forms. By its nature, user experience testing is manual and exploratory.
Don’t confuse user experience testing (UX) with user acceptance testing (UAT). As discussed earlier, UAT is a testing level that verifies that a given application meets requirements. For example, imagine that shortly after an update is released to a shopping website, the site receives many complaints that customers are unable to save items to a wish list. However, UAT verified that pressing the “wish list” button correctly added items to the wish list. So, what’s wrong? UX testing might have revealed that the wish list button was improperly placed on the screen, making it difficult for shoppers to find.
You can conduct this GUI testing method at any point in the development phase where user feedback is needed. It is not necessary to have a completed application prior to involving users. For example, focus groups can respond to screen mockups or virtual walk-throughs of an application early in development.
User Experience Testing Benefits
UX testing provides the essential perspective of the end-user, and therefore it can identify defects that may be invisible to developers and testers due to their familiarity with a product. For example, a few years ago, a leading web-based e-mail provider developed a social sharing tool. This tool was beta-tested by thousands of the provider’s own employees but not by end-users prior to its initial release. Only after the product was released into production did the provider begin receiving end-user feedback, which was overwhelmingly negative due to privacy concerns. Early UX testing would likely have revealed these concerns and saved the provider millions of dollars in development costs.
User Experience Testing Challenges
UX testing requires the identification and recruitment of user testers that accurately represent the target user base, such as novices, experienced users, young people, older adults, etc. While it is not necessary to have a very large group of users, it is important to cover the expected user personas. Recruiting user testers may be a time-consuming and potentially costly process. Although it is possible for user experience testing to be done by user proxies such as product owners, it can be difficult for them to set aside their knowledge of the AUT and fully step into the role of an end user. Finally, if UX testing is limited to beta testing, this occurs very late in the development cycle when it is expensive to find and fix defects.
What Is the Best GUI Testing Technique?
Scripted testing, exploratory testing, or user experience testing: which is the best GUI testing method for your situation? All decisions regarding testing should seek to maximize the value of an application for its users, both by detecting defects and by ensuring functionality and usability. In most cases, achieving this goal will require a combination of test techniques. Choosing the particular combination that best fits your application and development environment is done in the test planning phase, described in the next section.
Shift left testing
How to Write a GUI Test Plan
A GUI testing plan sets the scope of a test project. Before writing test cases, it is important to have a test plan that identifies the resources available for testing and that prioritizes areas of the application to be tested. Given this information, a testing team can create a test charter for exploratory testing, test scenarios, test cases, and test scripts for scripted testing.
The test plan defines key information including:
- Anticipated dates of testing
- Required personnel
- Required resources, such as physical hardware, virtual or cloud-based servers, and tools such as automation software
- The target test environments, such as desktop, mobile devices, or web with supported browsers
- The workflows and events of the AUT to be tested, as well as the AUT’s visual design, usability, and performance.
- Planned testing techniques, including scripted testing, exploratory testing, and user experience testing.
- The goals for testing, including criteria for determining the success or failure of the overall testing effort.
Test plans can be text documents, or you can use a test management tool to develop the test plan and to support analysis and reporting. There are many such tools available, including free server- and cloud-based tools. In the absence of a formal management tool, it is not uncommon to use a spreadsheet to track the progress of testing.
Remember that a GUI test plan is not a full system test plan, which would test other aspects of an AUT such as load testing, security, and backup and recovery.
Identify Areas for GUI Testing
There are several ways to identify the areas of the user interface to test. If specification documents are available, this is a good place to start. If specification documents are unavailable or incomplete, a useful approach is to conduct a brainstorming or concept mapping session to determine areas for GUI testing.
The list below can help you start a brainstorming session:
- Visual Design
- Functionality
- Performance
- Security
- Usability
- Compliance
The sample concept map below shows the result of a brainstorming session for a generic application and includes GUI events such as add, edit, delete, save, and close. To create a concept map, testers apply heuristics: their knowledge of the AUT combined with general testing principles.
Partial concept map for GUI testing
For example, to verify the navigation of a cloud-based application, plan tests such as the following:
Sample areas to test for web navigation
- Compatibility with all common browsers
- Proper functioning of the page when the user clicks the back button or the refresh button
- Page behavior after a user returns to the page using a bookmark or their browser history
- Page behavior when the user has multiple browser windows open on the AUT at the same time.
Prioritizing Test Cases for Risk-Based GUI Testing
Because resources for testing are often limited, it can be helpful to prioritize areas to test. Risk-based testing uses an analysis of the relative risk of potential defects to select the priorities for testing. Risk analysis is completed using a matrix similar to the one below. In this matrix, the Frequency column describes how often a user might encounter a potential defect, which includes both how visible the function is and how often it is used. Each of the Impact columns describes the effect of the defect on the user. The combination of frequency and impact determines the level of risk.
Risk Assessment Matrix
For example, let’s assume that the password reset process does not work at all. Once a user encounters it, the frequency is “constant” and the effect is “catastrophic” as the user is locked out of the application. Therefore, testing the password reset process is a critical priority. If the GUI testing team determines that there is only enough time to inspect critical and high-priority events, then medium events could be addressed through exploratory testing, while low-priority events may not be tested at all.
Planning Regression Testing
As mentioned earlier, regression testing helps ensure that new defects haven’t been introduced to previously-working code. Using a test automation tool such as Ranorex Studio can significantly increase the number of regression test cases that can be completed in a testing window. However, even with automation, it may be impractical to repeat all of the previous test cases for a new release. The most important regression test cases to perform are listed below.
The most important regression test cases are those which:
- Have the highest level of risk
- Provide the greatest coverage of code
- Are likely to uncover the greatest number of defects, based on how many defects each test case identified in prior rounds of testing
To avoid wasting time and effort on an application that is not ready for full testing, a test plan may also include smoke testing and sanity testing.
Smoke and sanity testing
- Smoke testing checks the basic functionality of an application. For example, smoke testing would verify that the AUT can start and that users can log on. Smoke testing is shallow, because it does not test any one part of the system in depth, and it is also wide because it covers as much of the major functionality as possible. The name comes from the practice of turning on a new piece of hardware to see if it catches fire. If it does not, additional testing can proceed.
- Sanity testing examines just the new or modified code to ensure that it not causing any major problems and that it meets specifications. Compared to shallow and wide smoke testing, sanity testing is narrow and deep.
Both smoke and sanity testing are often performed by developers before more rigorous reviews by software testers.
How to Write GUI Test Scenarios
A test scenario is a brief statement of how an application will be used in real-life situations, such as “the user will be able to log in with a valid username and password.” GUI testing scenarios can be written from development documents such as requirements, acceptance criteria, and user stories. In the absence of such documents, scenarios can be developed in consultation with developers and customers/customer representatives.
Scenarios can guide exploratory testing, giving testers an understanding of a GUI event to test, without restricting them to a specific procedure. Scenarios are also increasingly popular in agile environments, as it is much faster to create a brief scenario than to write out a full test case.
Scenarios are not required to create GUI test cases but help guide their development. If scenarios are used in scripted testing, then they serve as the base from which test cases can be developed, as shown in the diagram on the right.
Test scenarios, if used, guide the development of test cases and test scripts.
For example, the “log in” scenario above could have test cases for GUI events such as the following:
- User enters a valid username and password
- User enters invalid username
- User enters valid username but invalid password
- User tries to reset the password
- User tries to copy the password from or to the password field
- User presses the help button
How to Write GUI Test Cases
To write a GUI test case, start with a description of a GUI event to be tested, such as a login attempt. Then, add the conditions and procedures for executing the test. Finally, identify the expected result of the test and the criteria for determining whether the test succeeds or fails.
Whether to write general or detailed procedures depends on factors such as:
- The experience level of the testers, both with GUI testing in general and with the specific application being tested. Less-experienced testers may need more detailed procedures.
- How often the user interface changes. If an interface changes frequently, maintaining detailed procedures requires more effort.
- How much freedom end-users will have when navigating through the application. If users will have a lot of freedom, you could write procedures to cover all of the possible navigation paths, or else rely on the ability of the testers to anticipate the random paths that users might take.
If testers need only general procedures, these could appear in the test case itself. If GUI testing teams need detailed procedures, putting these in a separate test script may help make your tests more maintainable.
What to Include in a GUI Test Case
The most basic information in a test case is a description of the GUI event to be tested, the conditions for executing the test, and the expected result. To make test cases easier to manage, it can be helpful to include additional information such as links to requirements documents and/or defect tracking systems.
For example, the “valid username and password” test case above could contain information such as:
| Test Case ID | a unique identifier for the test script. |
| Title | the title of the test script, such as “User enters a valid username and password, maximum length.” |
| Scenario/ Requirement ID | link to the unique ID for the test scenario or requirements document, if applicable. |
| Priority/Risk Level | critical, high, medium, low. |
| Technique | scripted, exploratory, UX |
| Description | a brief explanation of the test case, such as “when the user is not already logged on, ensure that the user can log on with any valid character combination for the username and password, including special characters. Ensure that the password is hidden unless the user chooses to make it visible |
| Data Source* | external spreadsheet or database containing combinations of usernames and passwords to test |
| Procedure* | list of steps for the tester to follow when performing the test |
| Expected Result* | success, application main window appears |
| Actual Result* | completed by tester after testing |
| Status | pass/fail/blocked status of the test case. |
| Defect Cross-Reference* | if a defect is found, enter the code from the defect tracking system here, to connect the test case with the defect. |
*If you choose to write test scripts, this information appears in the test script rather than the test case. See below
Best Practices for Writing GUI Test Cases
Applying “best practices” to your test case design can help improve the quality of your tests. Here are some suggestions to make GUI testing easier to maintain and execute.
Separate Test Data from Test Cases
Separating test data from your test cases will make them easier to maintain. For example, the “valid username and password” test case should not include the actual username and password data values. Instead, these data values should be kept in a spreadsheet or database – whether the GUI tests are performed manually or with the help of test automation software. Likewise, it is also a good idea to separate information about the environment from the test case. Should the team decide to test on a new platform, it will not be necessary to change the test case itself.
Keep Test Cases Modular
To the extent possible, keep test cases modular so that they can be performed in any order. This more closely mimics the actual user experience, because users don’t always go through an application in the order that developers expect. In the case of a cloud-based shopping application, don’t write one large test case for buying an item. Instead, create separate test cases for events such as searching for items, adding items to a cart, deleting items from a cart, and updating the quantity of items in the cart. This GUI testing practice will make it easier to test the combination of events such as a user going back to the search feature after adding several items to the cart, rather than going straight to checkout.
Write Positive and Negative Test Cases
Be sure to write both positive test and negative test cases. A positive test case verifies the behavior of the AUT when a user enters valid data, while a negative test case verifies the response of the AUT to invalid data. For example, the cloud-based shopping application might have the following test cases:
- Positive test case: enter a valid credit card in the payment field. The test succeeds if the AUT accepts a valid payment method.
- Negative test case: enter an invalid credit card in the payment field. The test succeeds if the AUT gives the specified error message.
Use Testing Heuristics
When creating data for GUI testing, it is useful to draw on heuristics. For example, create test data for the maximum and minimum values in a data field. Or, when testing queries against a database, have tests for a query that returns zero rows, one row, or multiple rows. For more examples of testing heuristics, see the Test Heuristics Cheat Sheet by agile testing expert Elisabeth Hendrickson.
How to Write GUI Test Scripts
A test script provides a clearly defined procedure for a tester to follow. The test script may include information such as the following:
| Test Script ID | a unique identifier for the test script. |
| Title | the title of the test script, such as “User enters a valid username and password, maximum length.” |
| Test Case ID | a link to the unique ID for the test case. |
| Test Setup | the requirements for the test environment; could be stored separately in a test data spreadsheet. |
| Data | either the literal values for the tester to enter; or a link to an external spreadsheet or database containing combinations of usernames and passwords to test. |
| Procedure | the step-by-step instructions for the tester, such as the following example:
|
| Actual Result | completed by tester after testing. |
| Status | pass/fail/blocked status of the test script. |
| Defect Cross-Reference* | if a defect is found, enter the code from the defect tracking system here, to connect the test case with the defect. |
Create sufficient test scripts to verify the most common paths that users will take through the AUT.
Start a Free Trial of Ranorex Today
Ranorex Studio offers a set of automated GUI testing tools that will make your test process faster and more effective. Start a free trial to start GUI testing today.
Related Posts:
Model-Based Testing with Ranorex DesignWise
Model-based testing (MBT) has emerged as a powerful strategy for maintaining high standards of quality in an efficient and systematic manner. MBT transforms the development process by allowing teams to derive scenarios from models of the system under test. These...
What Is OCR (Optical Character Recognition)?
Optical character recognition technology (OCR), or text recognition, converts text images into a machine-readable format. In an age of growing need for efficient data extraction and analysis processes, OCR has helped organizations revolutionize how they process and...
Support Corner: API Testing and Simple POST Requests
Ranorex Studio is renowned for its robust no-code capabilities, which allow tests to be automated seamlessly across web, mobile, and desktop applications. Beyond its intuitive recording features, Ranorex Studio allows custom code module creation using C# or VB.NET,...