Efficient product development is always about trade-offs. One of the first considerations in undertaking any test automation project is where to focus your efforts. Resources are invariably limited, so which efforts will give you the greatest payoff? Which test cases will give you the highest return on the time invested? This article provides recommendations for three types of test cases: those to automate, those that will be challenging to automate, and those that shouldn’t be automated at all.
In principle, any software test can be automated: humans who understand the requirements for an application can create tests that express those requirements. Wise testers always ask, though, whether a particular test will cost more to develop and maintain than it will save in the effort of manual testing. To get the best return on your effort, focus your automation strategy on test cases that meet one or more of the following criteria:
Note that this is different from the stability of the implementation. Once functionality has been settled, it’s particularly valuable to have good automated tests if the development team continues to experiment with alternative implementations.
To learn more about regression testing, refer to the Ranorex Regression Testing Guide.
A smart test team labels and actively maintains different categories of tests. A test of a specific functionality might move in and out of the smoke test suite at different times during the lifecycle of the application. When a particular login method is widely used, it deserves to be a smoke test. If later it’s deprecated in favor of a different method, it might safely be moved away from the smoke tests. Similarly, a test that once was too time-consuming to be a smoke test can become a smoke test if it’s accelerated enough to fit in CI/CT (“CT” abbreviates “Continuous Testing”).
Load and other performance tests often are too expensive and time-consuming to execute with each commit. This just means that they need their own schedule for execution, one that doesn’t slow the CI cycle.
Along with other reasons, it’s good to automate cross-browser and cross-device tests because humans typically do not perform well in this tedious and repetitive role. Automation has proven far better at spotting environment-specific problems such as browser incompatibility.
Cross-Technology | Cross-Device | Cross-Platform
The following types of test cases are more difficult to automate. That doesn’t mean that they shouldn’t be automated – only that these test cases will have a higher cost in terms of time and effort to automate. Whether a particular test case will be challenging to automate varies depending on the technology basis for the AUT (application under test). If you are evaluating an automation tool or doing a Proof of Concept, be sure that you understand how the tool can help you overcome these difficult-to-automate scenarios.
This last point is so important it bears repeating: a test might be worth automating even though it’s difficult to automate. Sometimes the difficulty of automation of a test reflects that the corresponding manual test is particularly time-consuming or error-prone or sensitive. That case probably means that such a test is especially valuable to automate, or perhaps redefine to be less expensive.
One general response to difficult automations is to seek help, whether the leverage of a high-quality test framework that solves hard problems, or counsel from fellow professionals who’ve faced similar problems.
Automated tests can fail when an expected response is not received. It’s important to handle waits rigorously so that a test doesn’t fail just because the system is responding slower than normal. However, you must also ensure that a test does fail in a reasonable period of time so that the entire test suite is not stuck waiting for an event that will never happen. Issues around synchronization and waits are particularly important when comparing test frameworks. To learn how to configure waits in Ranorex automated tests, refer to the description of the Wait for action in the Ranorex User Guide.
To minimize these types of issues, make your test cases as modular and independent of each other as possible, and then manage the workflow with a keyword-driven framework. Measure source coverage accurately to minimize the count of unexercised lines of source.
There are some types of tests where automation may not be feasible or advisable. This includes any test where the time and effort required to automate the test exceeds the potential savings. Plan to perform these types of tests manually.
To ensure that you achieve your automation goals, focus your automation efforts on the right test cases. And be sure to build in time for exploratory testing and UX/usability testing – by their nature, these types of tests can’t and shouldn’t be automated.
To help determine whether or not to automate a particular test case, you can use the Test Case ROI Calculator spreadsheet. This simple spreadsheet compares the estimated time and costs to automate a test case vs. the time and costs to execute the same test case manually; it is not designed to determine the ROI of a test automation project as a whole. With a little up-front analysis, though, you can make tactical decisions about individual tests that yield the best possible results for your project: the biggest bang of completed, informative tests for the buck of the effort invested in testing.
Strategies for a Successful Automation Project: Learn how to ensure that your automation project accomplishes your goals.
Cross-Technology | Cross-Device | Cross-Platform
Ranorex Studio is renowned for its robust no-code capabilities, which allow tests to be automated seamlessly across web, mobile, and desktop applications. Beyond its intuitive recording features, Ranorex Studio allows custom code module creation using C# or VB.NET,...
It’s difficult for any IT organization to execute DevOps effectively without test automation. However, it’s often easier said than done. Overcoming the challenges of automated software testing can end up slowing down product delivery and impacting quality, the exact...
There are more and more Android testing tools available for mobile app developers. These are our favorites for performance, accessibility, and security.
