Test Automation Best Practice #9: Plan E2E Testing

Jun 27, 2018 | Best Practices, Test Automation Insights

Best practices in test automation, end-to-end testing

Introduction to End-to-End Testing

End-to-end testing (E2E) examines the real-world scenarios of an application from start to finish, touching as many functional areas and parts of the application’s technology stack as possible. Compared to unit tests, which are narrow in scope, E2E tests have a broad scope, and so are sometimes called “broad stack” or “full stack” tests. E2E tests focus on validating the workflows of an application from the perspective of the end-user, which makes them highly valued by management and customers. E2E testing is usually performed last in the testing process, following lower-level unit, integration, and system testing.

Despite their value, automated E2E tests can be complex to build, fragile, and challenging to maintain. As a result, a common approach is to plan a smaller number of E2E tests than unit and integration tests, as shown in the test automation pyramid. E2E testing is conducted in as realistic an environment as possible, including the use of back-end services and external interfaces such as the network, database, and third-party services. Because of this, E2E testing can identify issues such as real-world timing and communication issues that might be missed when units and integrations are tested in isolation.

E2E Example

Assume that you are testing a web shopping platform that requires a third party to validate payment details. This application might contain E2E tests such as the following:

User logs on, searches for an item, puts the item in the cart, selects payment and shipping details, and then checks out and logs off.
Use logs on, searches for an existing order that has been shipped, reviews the tracking information, and receives a detailed response on the delivery of the order, then logs off.
User logs on, searches for an existing order that has been shipped, requests a return of the order, receives a shipping label to return the item, and logs off.
User logs on, opens their account information, adds a new payment type, receives verification that the payment type is valid, and logs off.

These tests will access third-party services such as payment verification and shipment tracking, as well as one or more databases for customer information, inventory, orders, and more.

Solve Your Testing Challenges
Test management tool for QA & development

Download Free Trial

Best Practices for E2E Testing

A typical E2E test can be complex, with multiple steps that are time-consuming to do manually. This complexity can also make E2E tests difficult to automate and slow to execute. The following practices will help manage the costs of automated E2E testing while maintaining the benefits.

Keep an end-user perspective

Design E2E test cases from the perspective of an end user, with a focus on the features of the application rather than its implementation. If available, use documents such as user stories, acceptance tests, and BDD scenarios to capture the user’s perspective.

Limit exception testing

Focus E2E tests on high-use “happy path” or “golden path” cases that capture typical use scenarios like the ones listed for the web shopping platform described above. Use lower-level unit and integration tests to check bad path/sad path exception cases, such as a user attempting to order more of an item than is currently in inventory, or returning an item past the allowable return date.

Apply risk analysis

Given the relative expense of performing E2E tests manually or automating them, concentrate on your application’s high-risk features. To determine a high-risk feature, consider both how likely a failure is to happen, and the potential impact that it would have on end users. A risk assessment matrix is a useful tool in identifying risk. Read more about risk analysis in the Ranorex Testing Wiki.

Test in the right order

When a single unit test fails, it’s relatively easy to figure out where the defect occurred. As tests grow in complexity and touch more components of an application, the increase in potential points of failure make it harder to debug them when a failure occurs. Running your unit and integration tests first allows you to catch errors when they are relatively easy to resolve. Then, during E2E testing, complete your critical smoke tests first, followed by sanity checks and other high-risk test cases.

Manage your test environment

Make your environment setup process as efficient and consistent as possible. Document the requirements for the test environment and communicate them to system administrators and anyone else involved in the setup of the environment. Include in your documentation how you will handle updates to the operating system, browsers, and other components of the test environment to keep it as similar as possible to the production environment. One solution may be to use an image backup of the production environment for testing purposes.

Separate test logic from your UI element definitions

To make your automated E2E tests more stable, separate the logic of your tests from the UI element definitions. Use an object repository or a page object pattern to avoid having your test logic interact directly with the user interface. This makes your tests less likely to fail due to changes in the structure of the UI. Ranorex Studio includes an object repository for separating automated tests from UI element definitions, or you can use the page object pattern when testing web applications with Selenium.

Handle waits for UI elements appropriately

Don’t allow an E2E test to fail unnecessarily when waiting for a page to load or a UI element to appear on the screen. In Ranorex Studio, you can add an action that waits for a UI element to exist, for a specified period. If that period is exceeded, then the test fails. The wait time should be at least as long as the normal time that it should take for the UI element to appear. But don’t make it too long. Excessively long wait times may indicate a problem with the application, interfaces or environment, and are annoying to end-users. In addition, allowing long wait times in your automated tests can also affect the overall execution of your E2E testing. In general, set a wait time that is just little longer than the normal time it takes for a UI element to appear.

Choose the right devices

For mobile devices, concentrate physical testing on the most popular versions of iOS and Android, and then use simulators/emulators to provide coverage for less popular versions. Test on both WiFi and cell carrier connections at various speeds from 3G to LTE.

Optimize your setup/teardown process

Use setup/teardown processes so that your test environment is always ready to go. Create your test data using automated scripts, and clean it up afterward so that the test environment is restored to its original status and is ready for the next round of testing.

Conclusion

It’s important to plan manual and exploratory testing as part of your E2E testing, to address difficult-to-automate aspects such as usability and user experience. And, to ensure that you have a complete and well-balanced set of tests, include automated performance and load testing, which will be covered in the final article in this series.

All-in-one Test Automation

Cross-Technology | Cross-Device | Cross-Platform

Download Free Trial

Support Corner: API Testing and Simple POST Requests

Ranorex Studio is renowned for its robust no-code capabilities, which allow tests to be automated seamlessly across web, mobile, and desktop applications. Beyond its intuitive recording features, Ranorex Studio allows custom code module creation using C# or VB.NET,...

The Top 10 Test Automation Challenges

It’s difficult for any IT organization to execute DevOps effectively without test automation. However, it’s often easier said than done. Overcoming the challenges of automated software testing can end up slowing down product delivery and impacting quality, the exact...

7 Best Android Testing Tools

There are more and more Android testing tools available for mobile app developers. These are our favorites for performance, accessibility, and security.

Support

Education

Blog

Testing Solutions

Platforms

Integrations