The Pros and Cons of End-to-End Testing

The goal of end-to-end (E2E) testing is to thoroughly review the user experience from start to finish. Examining real-world scenarios for an application seems essential. However, E2E testing is sometimes viewed as too cumbersome. It might be overlooked in favor of developer testing, or unit and integration testing. E2E testing, despite its challenges, is important to validate the workflow of an application from the end user’s perspective.

Let’s start with the positive aspects of end-to-end testing.

The pros of end-to-end testing

With its broad scope, end-to-end testing also is known as “broad-stack” or “full-stack” testing. E2E offers the full picture. Here are some of the benefits of E2E testing:

• Tests from a user’s perspective: while unit tests verify each component in isolation, end-to-end tests examine the application from the perspective of an end-user. This can uncover defects that aren’t apparent in unit testing.
• Verifies workflows: E2E tests can validate the business logic of an application.
• Expands test coverage: E2E tests can verify that all of an application’s dependencies work correctly together, including third-party code.
• Reduces the number of errors found in production: Adding end-to-end testing to a test suite can help reduce the chance of finding defects after deployment to production.

The cons of end-to-end testing

As mentioned above, end-to-end testing can be viewed as cumbersome. Here are some of the perceived negatives with E2E testing:

• Slow execution: Testing that verifies workflows through the UI can take a long time to run.
• Fragile/flaky tests: E2E tests may require significant maintenance and troubleshooting.
• Lack of available test environment: Re-creating a testing environment for a real-world scenario might be difficult.
• More difficult to debug: When an E2E test fails, more research might be needed to find and fix the issue than with a more granular unit or integration test.

How much end-to-end testing is necessary?

Faced with the question of whether to perform end-to-end testing — and how much to perform — a common approach is to turn to the well-known Test Automation Pyramid, shown below. Following this model, the bulk of automated tests would be unit tests: small, quick tests that are executed below the level of the user interface. The middle layer is composed of API/integration tests that validate several components of an application together, such as database services, APIs, and third-party services. Since E2E tests are typically performed trough the UI, they are placed at the top of the pyramid.

To put a numeric value on each of the layers, in a 2015 article, Mike Wacker of Google described the test automation pyramid as a 70/20/10 split with 70% unit tests, 20% integration tests, and 10% automated E2E tests.

However, the test automation pyramid represents an idealized scenario. It’s not appropriate for every situation. For example, in a complex ERP application that includes many third-party components that can’t be unit tested, it might make sense to turn the pyramid upside down, and conduct mostly E2E tests.

Raj Subrameyer has suggested an alternative context-based model based on a hub-and-spoke diagram, with automated testing at the center. This removes the concept of “levels” and allows teams to think more freely about the types of tests that might be appropriate for a given context.

How to get started with end-to-end testing

So, the question becomes how to overcome E2E testing challenges. One answer is automation. In fact, along with regression testing, E2E testing is one of the best practices for test automation.

Of course, automated testing itself has numerous benefits, including:

• Saves time for tests that are repeated for different cases
• Saves resources, especially as tests can be scheduled for off-hours when systems might otherwise be idle
• Delivers fast feedback from testing
• Provides easily-repeatable testing
• Enables testing in production

In a recent third-party survey, more than half of Ranorex users report automating their E2E testing.

Here is what one customer had to day about their strategy for automating end-to-end testing with Ranorex Studio.

For more information on end-to-end testing, read our article 10 Best Practices in Test Automation #9: Plan E2E Testing.