Test Automation Best Practice #7: Integrate with a CI Pipeline

Mar 15, 2021 | Best Practices, Test Automation Insights

Integrate with a CI Pipeline
Rapid application development practices such as Continuous Integration/Continuous Deployment (CI/CD), and DevOps have a common goal: small, frequent releases of high-quality, “working” software. Whether your development cycle is measured in weeks or days, integrated automated tests are essential to maintaining the pace of development.

Automated tests in a CI pipeline

The image below shows a typical CI pipeline. A developer checks out code from the shared repository in the version control system, such as Git, TFS or Subversion. Once code changes are complete, the developer commits the change back to the version control system, triggering a CI job. The CI server builds the application under test and triggers automated tests to verify whether the new code results in a good, “green” build. The results of testing are reported back to the entire team for a decision regarding deployment of the application. In a CD environment, the application is deployed automatically to the production environment.

Accelerate your testing with CI/CD integration
A continuous integration pipeline
Continuous integration with automated testing offers several benefits to organizations, including the following:

  • Developers get fast feedback on the quality, functionality, or system-wide impact of their code changes, when defects are easier and less expensive to fix.
  • Frequent integration of small changes reduces the volume of merge conflicts that can occur when several developers are working on the same application code, and makes merge conflicts easier to resolve when they do happen.
  • Everyone on the team has a clear understanding of the status of the build.
  • A current “good build” of the application is always available for testing, demonstration, or release.
  • Frequent releases make for good practice in a successful release process. Rather than hazards to be avoided, updates become routine events in a healthy software development lifecycle (SDLC).

Recommendations for automated testing in a CI pipeline

The recommendations below focus on test automation in a CI pipeline, some of which overlap the best practices for the CI process itself. Read more about best practices for a CI process in the Wikipedia article on Continuous Integration.


Use source control for your automated tests

The Twelve-Factor App is a methodology for building software-as-a-service that’s widely-regarded as authoritative for general best practices in modern coding. Among all these, the very first is, one codebase tracked in revision control.

That’s the starting point for all automation, integration, and release-management efforts: maintain your automated tests under revision control, and in fact in the same repository as your code. Good control over test sources pays off in many ways, first among these being to make it easier to match correctly the version of a test to the version of the source code. Ranorex Studio integrates with popular solutions for source control including Git, Microsoft Team Foundation Server, and Subversion.

An independent quality assurance (QA) team responsible for specialized tests might maintain sources in a separate repository. In all cases, though, the main points remain:

  • sources, including test sources, need to be under revision control
  • programming source and CI test source need to be coordinated
Icon unit testing

Don't rely solely on unit tests

Unit testing in an individual developer’s local environment doesn’t tell you enough about how that code will work once it is introduced to the production application. Integration of new or revised code may cause a build to fail for several reasons. For example, changes made by another developer may conflict with the new code, or there may be differences between the developer’s local environment and the production environment. Therefore, it’s important to run integration tests, regression tests and high-priority functional UI tests as part of the build verification process.

Notice, too, the distinction between “unit testing in an individual developer’s local environment”, and the “green” build mentioned above. The latter is critical: a standardized, repeatable criterion with a definite relationship to a consistent test environment. Reliance on developers to submit what passes in a local environment introduces too much uncertainty.

With that standardized test environment in place, extension of CI from unit testing to integration tests (and more) is natural.

Icon self-testing build

Make your build self-testing

Component testing checks individual units of code. Component testing is often called unit testing, but may also be called module testing or program testing. Developers write and execute unit tests to find and fix defects in their code as early as possible in the development process. This is critical in agile development environments, where short release cycles require fast test feedback. Unit tests are white-box tests because they are written with a knowledge of the code being checked.
Icon refactor automated test

Refactor your automated tests

A build that takes a long time to complete disrupts the CI process. To keep your testing efficient, approach your automated testing code as you would the application code itself. Regularly look for redundancies that can be eliminated, such as multiple tests that cover the same feature or data-driven tests that use repetitive data values. Techniques such as boundary value analysis and equivalence partitioning can help reduce your data-driven testing to just the essential cases.

Automated tests frequently bottleneck on network operations, or, more generally, use of external resources. When developers notice tests slowing, one of the remedies your refactoring can take is to mock those resources for fast results.

Icon stopwatch

Keep your build fast

It’s essential that build tests complete as quickly as possible so that developers aren’t discouraged from committing frequently. To keep the process fast, trigger the minimum automated tests required to validate your build.

Due to their more complex nature, integration tests are usually slower than unit tests. Run your smoke and sanity tests first to rapidly identify a broken build before spending time on additional tests. If your team merges frequently, it may be more efficient to run integration tests only for daily builds rather than every merge.

Run a full regression only when necessary, such as in preparation for deployment to the production system. For example, Ranorex Studio supports the use of run configurations to run a subset of tests, such as smoke tests or tests just for features or modules of the application that have changed. Exclude from the build test set any low-priority regression test cases that haven’t found a defect in several test cycles.


Test in the right environment

To minimize the chance of test failures due to issues such as incorrect O/S version or missing services, test in an environment that is stable. Ideally, you will have an isolated test platform that is dedicated solely to testing. Your test environment should also be as identical as possible to the production environment, but this can be challenging. Realistically, it may be necessary to virtualize or mock certain dependencies such as third-party applications. In complex environments, a virtualization platform or solution such as Docker containers may be an efficient approach to replication of the production environment.
Icon parallel testing

Test in parallel

Speed is essential in a CI/CD environment, as two previous sections already mentioned. Quick return of results pays off enormously in making the most of developers’ flow. Save time by distributing your automated tests on a Selenium grid or running them in parallel on multiple physical or virtual servers. As mentioned earlier in this series, keep your automated tests as modular and independent of each other as possible so that you can test in parallel.
Support for UI controls

Include functional UI and exploratory testing

It takes a combination of automated testing approaches to confirm that your application is ready for deployment to the production environment. In addition to your automated unit and integration tests, include automated user interface tests to verify core functionality, check common user paths through the application end-to-end and validate complex workflows. Exploratory testing can uncover defects that automated tests miss.
Icon verify deployment

Verify your deployment

After deploying the new build to the production environment, run your smoke tests in the production environment as a quick check to ensure a successful deployment.

To learn more about how to integrate Ranorex Studio tests in your CI pipeline, read our blog article Integrate Automated Testing into Jenkins. While this article focuses on Jenkins, Ranorex tests can be triggered from any CI server process, including Bamboo and TeamCity.

All-in-one Test Automation

Cross-Technology | Cross-Device | Cross-Platform

Related Posts:

Support Corner: API Testing with Ranorex Studio and a GET Request

Support Corner: API Testing with Ranorex Studio and a GET Request

Ranorex is a powerful no-code tool that automates web, mobile, and desktop application testing. In addition to the powerful no-code recording, you can utilize the Ranorex API to create code modules in C# or VB.NET. That’s beneficial because it can increase the...

How Does Ranorex Rank on Review Sites?

How Does Ranorex Rank on Review Sites?

If you're part of a software team, you know how challenging and rewarding user interface (UI) testing can be. After all, it's a time-consuming process that involves technical knowledge and experience. But the feeling of a successful UI test—when your application runs...

Exploring 5 Types of Usability Testing

Exploring 5 Types of Usability Testing

  Usability testing gives you valuable insights into how real users interact with your product. You can use the reactions and feedback to refine your product to ensure it’s effective and enhances the user’s enjoyment. We’ll explore different types of usability...