Time is money, particularly in IT. Every bug, every missed date, every slow-moving process takes time away from getting software out the door and affects the bottom line. Hence, organizations are always looking for ways to improve efficiency: to do more in less time.
DevOps has proven key to creating the efficiencies needed to stay competitive in an increasingly demanding business environment. History shows that following the fundamental principles of DevOps — cross-functional teamwork, automation wherever possible, fearless communication, and treating infrastructure as code — works! However, achieving a sound DevOps practice doesn’t happen by magic or overnight. Planning, commitment, and hard work are required. This is especially true when it comes to implementing test automation in a DevOps pipeline. It’s not a one size fits all undertaking. Rather, effective implementation requires that companies pay attention to 3 essential considerations. These considerations are:
- Build flexibility into the DevOps pipeline.
- Segment the testing process across the pipeline.
- Understand that not all testing can be automated.
Allow me to elaborate.
Build flexibility into the DevOps pipeline
A typical DevOps deployment pipeline is divided into stages, with each stage having a distinct purpose, as shown below:
- User Acceptance
The purpose of a DevOps pipeline
The image above shows just one possible DevOps pipeline. In practice, the number, name, and implementation of stages in a DevOps pipeline can vary. Some organizations have three stages while others have six or more stages. Regardless of the number or names of the stages, the important thing is that these are consistent and that the purpose of each stage is well understood and supported on a company-wide basis.
The purpose of a DevOps deployment pipeline is to create stages through which code moves and is inspected and tested. Testing becomes more stringent at each stage and thus reliability and quality increases.
During the first stage — Development — the rate of change in the code is high. New features are added and bugs are fixed. Change can happen on an hourly basis. By the time the application escalates to the final Release stage, the rate of change diminishes to a near standstill as the code becomes more thoroughly tested. By the Release stage, the code is hardened and ready to be published to the public.
Each stage in the DevOps pipeline has a purpose. Testing is performed according to the scope and purpose of each stage in the process. The pipeline is predictable. Code moves from one stage to another in sequence. That’s how it’s supposed to go … until it doesn’t.
Why flexibility is critical
No matter how well defined a company’s DevOps pipeline is, there will be exceptions that need to be accommodated. Usually, the exception is a hotfix. A hotfix is a piece of mission critical code that needs to be released as soon as possible, almost immediately.
Hotfixes happen all the time. When they do, some companies panic, circumvent the pipeline and send code right from development to release while hoping all will work out for the best. Companies more adept with DevOps will have the benefit of foresight and understand that there are times when the DevOps pipeline will need to accommodate episodes of code escalation outside of the usual deployment process. These companies build flexibility into their DevOps pipelines. They actually have a process by which to fast track code to release while not losing the benefits that a formal pipeline provides.
Such fast-tracking might mean creating a “side stage” in which the hotfix code is subjected to intense, controlled testing outside of the usual deployment sequence. Then, after the hotfix is released, a flexible deployment process will downstream merge the fast-tracked code back into the usual stages in a controlled manner.
Controlled, downstream merging in response to a hotfix is just one technique for providing flexibility in the DevOps pipeline. Each company will have its own techniques for addressing change that falls outside the usual deployment process. The trick is to be able to implement flexibility into the DevOps deployment process in ways that are controlled and safe.
When it comes to creating a viable DevOps deployment pipeline that is built to last, a flexible system will go the distance better than one that is rigid.
Segment the testing process across the pipeline
An important part of the DevOps pipeline is the ability to test code automatically once the code enters a particular stage of the deployment process. It just makes sense. Automated testing provides more efficiency than testing manually. However, efficiency can be compromised when automated testing is conducted in an arbitrary manner. Remember that each stage in the DevOps pipeline has a purpose. Conducting tests that fall outside the purpose of the stage is not a wise use of time.
The purpose of testing in the Development stage is to ensure that the application works according to expectation at the source code level. Typically such testing is accomplished by exercising the unit tests that ship along with the source code.
Component and functional testing take place when the code is built into components and the components interact with each other. Such testing usually takes more time than unit testing, despite being an automated undertaking. Component and functional tests are typically executed in the Staging part of the DevOps deployment pipeline.
Performance testing, which can involve creating millions of virtual users to go up against the application in a variety of network configurations can be conducted at the later part of Staging or in User Acceptance. Performance testing takes more time than component and functional testing. And then there’s Security testing, which can take just as long as Performance testing.
What’s important to understand is that just because you can conduct all tests at any stage of deployment, it doesn’t necessarily follow that you should. Tests take time. Running all tests, in every deployment stage can create time-consuming redundancy. The trick is to use testing time wisely to achieve the highest degree of quality possible. Thus, test planners will do well to work with DevOps personnel to determine the best places in the DevOps pipeline to conduct testing according to the purpose of the pipeline and the purpose of the testing. Thorough testing is critical to a successful DevOps pipeline. So is testing at the right time, in the right place. Segmenting automated testing properly along the DevOps pipeline will go a long way toward implementing a deployment process that is both efficient and reliable.
Understand that not all testing can be automated
Automated testing is essential in the philosophy and practice of DevOps. But, it’s important to realize that not all testing can be automated, particularly as you move into the later stages of the DevOps pipeline where the use cases can become more complex. Some tests really do need to be implemented by humans in real time. Performance and security testing scenarios that are notable use cases for manual testing usually involve a one-off situation that has a variety of special user personas or edge case operations. For example, performance testing under a special promotion scenario for an e-commerce site or doing security testing for a missile launch. These situations don’t happen every day and when they do happen, conditions tend to be unique to the testing time. In such cases, the time it takes to plan and do the manual testing might indeed be less than the time it takes to automate the process.
Granted, most testing can be automated and should be. But, companies will do well to understand that there is a time and a place for manual testing. They should plan on it. This means having the expertise on hand to design and implement the tests. Without proper planning, accommodating manual testing can incur an excessive cost. Planning for the use of manual testing will save time and money.
Putting it all together
Effective DevOps pipelines are not only flexible enough to support the special, episodic demands of the organization but are also structured enough to produce consistently reliable operations. Also, those organizations that have the foresight to segment their automated testing throughout the deployment pipeline, while also reserving the option to conduct manual testing when necessary, will experience greater value from their testing efforts.
The DevOps pipeline is an essential part of the DevOps way of life. It’s the place where all members of the team gather to create and deploy the software that makes the company move forward. And, as goes the pipeline, so goes the team. Paying attention to the considerations presented above will result in more efficient behavior all around.