Key Considerations for Test Automation in a DevOps Pipeline

Jul 27, 2021 | Integration, Test Automation Insights

Pipeline representing end to end testing

As organizations move to a continuous software delivery culture, DevOps has emerged as the leading development approach. A 2021 survey of over 282 technology leaders by slashdata showed that 77% would describe their deployment approach as “DevOps.”


In a DevOps environment, every bug, every missed date, and every slow-moving process that takes time away from getting software out the door represents a loss. As a result, organizations are always looking for ways to improve efficiency. But achieving a sound DevOps practice doesn’t happen by magic or overnight. Planning, commitment, and hard work are required. This is especially true when it comes to implementing test automation in a DevOps pipeline. It’s not a one-size-fits-all undertaking.

Test automation is widely regarded to be a an essential element of a DevOps pipeline. And companies have realized its importance now more than ever, as the pandemic forced them to work online, distributed, and with minimal resources.

In the 2021 Pulse Opinion Research survey, technology leaders identified test as a particular area of focus for improving their DevOps practice, while another 51% pointed to automations:

In this article, we are starting from the premise that test automation coupled with frequent deployments drives quality higher and helps release quicker. Here we discuss the main considerations for test automation in a DevOps pipeline.

1: Build in flexibility 

A typical DevOps deployment pipeline is divided into stages through which code moves and is inspected and tested, with each stage having a distinct purpose. Testing becomes more stringent at each stage and thus reliability and quality increases.

Because DevOps development is iterative, a common representation of a DevOps pipeline is a figure 8, as shown below:


The image above is just one possible DevOps pipeline. In practice, the number, name, and implementation of stages in a DevOps pipeline can vary. Some organizations have three stages while others have eight or more stages. Regardless of the number or names of the stages, the important thing is that these are consistent and that the purpose of each stage is well understood and supported on a company-wide basis.

During the early stages of development, the rate of change in the code is high. New features are added and bugs are fixed. Change can happen on an hourly basis. By the time the application escalates to the final release stage, the rate of change diminishes to a near standstill as the code becomes more thoroughly tested. By the deployment stage,  the code is hardened and ready to be published to the public.

Each stage in the DevOps pipeline has a purpose. Testing is performed according to the scope and purpose of each stage in the process. The pipeline is predictable. Code moves from one stage to another in sequence. That’s how it’s supposed to go … until it doesn’t.

No matter how well defined a company’s DevOps pipeline is, there will be exceptions that need to be accommodated. Usually, the exception is a hotfix. A hotfix is a piece of mission-critical code that needs to be released as soon as possible, almost immediately.

Hotfixes happen all the time. When they do, some companies panic, circumvent the pipeline, and send code right from development to release while hoping all will work out for the best. Companies more adept with DevOps will have the benefit of foresight and understand that there are times when the DevOps pipeline will need to accommodate episodes of code escalation outside of the usual deployment process. These companies build flexibility into their DevOps pipelines. They actually have a process by which to fast-track code to release while not losing the benefits that a formal pipeline provides.

Such fast-tracking might mean creating a “side stage” in which the hotfix code is subjected to intense, controlled testing outside of the usual deployment sequence. Then, after the hotfix is released, a flexible deployment process will downstream merge the fast-tracked code back into the usual stages in a controlled manner.

Controlled, downstream merging in response to a hotfix is just one technique for providing flexibility in the DevOps pipeline. Each company will have its own techniques for addressing change that falls outside the usual deployment process. The trick is to be able to implement flexibility into the DevOps deployment process in ways that are controlled and safe.

When it comes to creating a viable DevOps deployment pipeline that is built to last, a flexible system will go the distance better than one that is rigid.

2: Plan for testing at every stage

Having a proper test automation strategy is crucial for DevOps. You need to build a continuous process that is sustainable and repeatable as well as having built-in fast feedback mechanisms wherever possible.

An early step should be to scope the project for its automation needs. Since you cannot automate everything, prioritize the types of testing that will deliver the greatest results relative to the effort required to automate. This will include unit-level and integration tests at the lower levels of the Testing Pyramid in addition to regression tests at the UI level. A limited number of end-to-end tests executed after other tests have completed successfully can identify issues such as real-world timing and communication issues that might be missed when units and integrations are tested in isolation.

Planning will help ensure that you have the necessary expertise on hand to design and implement test automation. But no matter how many test cases you automate, some types of tests are difficult or impossible to automate, such as exploratory tests, unique cases like Captcha, one-time tests, or ad hoc tests. These tests are better done by humans. So a holistic approach to testing in DevOps will also include plans for human-centric, aka “manual” testing.

3: Segment the testing process

An important part of the DevOps pipeline is the ability to test code automatically once the code enters a particular stage of the deployment process. It just makes sense. Automated testing provides more efficiency than testing manually. However, efficiency can be compromised when automated testing is conducted in an arbitrary manner. Remember that each stage in the DevOps pipeline has a purpose. Conducting tests that fall outside the purpose of the stage is not a wise use of time.

The purpose of testing in the Development stage is to ensure that the application works according to expectations at the source code level. Typically such testing is accomplished by exercising the unit tests that ship along with the source code.

Component and functional testing take place when the code is built into components and the components interact with each other. Such testing usually takes more time than unit testing, despite being an automated undertaking. Component and functional tests are typically executed in the Staging part of the DevOps deployment pipeline.

Performance testing, which can involve creating millions of virtual users to go up against the application in a variety of network configurations, can be conducted in later parts of the process. Performance testing takes more time than component and functional testing. And then there’s security testing, which can take just as long as performance testing.

What’s important to understand is that just because you can conduct all tests at any stage of deployment, it doesn’t necessarily follow what you should. Tests take time. Running all tests in every deployment stage can create time-consuming redundancy. The trick is to use testing time wisely to achieve the highest degree of quality possible. Test planners will do well to work with DevOps personnel to determine the best places in the DevOps pipeline to conduct testing according to the purpose of the pipeline and the purpose of the testing. Thorough testing is critical to a successful DevOps pipeline. So is testing at the right time, in the right place. Segmenting automated testing properly along the DevOps pipeline will go a long way toward implementing a deployment process that is both efficient and reliable.

4: Automate more than just the tests

For DevOps pipelines to work effectively, there are more things to be done beyond just automation of tests. To make the process seamless and truly automated, we need to establish processes such as:

  • Automatic triggers for the build process
  • Triggers for test execution
  • Generation of reports
  • Success or failure criteria for builds
  • Automatic deployments to various test or production environments and sanity checks.

All of these steps need automation using different tools that will also be a part of your DevOps pipeline. Choose the tools wisely and experiment to see what tools fit the best and work well together in your context.

Putting it all together

Effective DevOps pipelines are not only flexible enough to support the special, episodic demands of the organization but are also structured enough to produce consistently reliable operations. Also, those organizations that have the foresight to segment their automated testing throughout the deployment pipeline, while also reserving the option to conduct manual testing when necessary, will experience greater value from their testing efforts.

The DevOps pipeline is an essential part of the DevOps way of life. It’s the place where all members of the team gather to create and deploy the software that makes the company move forward. And, as goes the pipeline, so goes the team. Paying attention to the considerations presented above will result in more efficient behavior all around.

For further recommendations on test automation in a DevOps pipeline, read our article 10 Best Practices in Test Automation #7: Integrate with a CI/CD pipeline.

All-in-one Test Automation

Cross-Technology | Cross-Device | Cross-Platform

Related Posts:

What Is Automation Testing?

What Is Automation Testing?

Test automation provides convenience and reliability to your software testing process. Learn more about how Ranorex Studio supports automated testing.

The Future of Artificial Intelligence in Test Automation

The Future of Artificial Intelligence in Test Automation

As technology continues to advance at an unprecedented rate, artificial intelligence (AI) is emerging as a transformative force across many industries. AI promises to revolutionize test automation. The end result? A more efficient, accurate, and reliable test...

Support Corner: API Testing with Ranorex Studio and a GET Request

Support Corner: API Testing with Ranorex Studio and a GET Request

Ranorex is a powerful no-code tool that automates web, mobile, and desktop application testing. In addition to the powerful no-code recording, you can utilize the Ranorex API to create code modules in C# or VB.NET. That’s beneficial because it can increase the...