Test Automation Best Practice #7: Integrate with a CI Pipeline

The Twelve-Factor App is a methodology for building software-as-a-service that’s widely-regarded as authoritative for general best practices in modern coding. Among all these, the very first is, one codebase tracked in revision control.

That’s the starting point for all automation, integration, and release-management efforts: maintain your automated tests under revision control, and in fact in the same repository as your code. Good control over test sources pays off in many ways, first among these being to make it easier to match correctly the version of a test to the version of the source code. Ranorex Studio integrates with popular solutions for source control including Git, Microsoft Team Foundation Server, and Subversion.

An independent quality assurance (QA) team responsible for specialized tests might maintain sources in a separate repository. In all cases, though, the main points remain:

• sources, including test sources, need to be under revision control
• programming source and CI test source need to be coordinated

Unit testing in an individual developer’s local environment doesn’t tell you enough about how that code will work once it is introduced to the production application. Integration of new or revised code may cause a build to fail for several reasons. For example, changes made by another developer may conflict with the new code, or there may be differences between the developer’s local environment and the production environment. Therefore, it’s important to run integration tests, regression tests and high-priority functional UI tests as part of the build verification process.

Notice, too, the distinction between “unit testing in an individual developer’s local environment”, and the “green” build mentioned above. The latter is critical: a standardized, repeatable criterion with a definite relationship to a consistent test environment. Reliance on developers to submit what passes in a local environment introduces too much uncertainty.

With that standardized test environment in place, extension of CI from unit testing to integration tests (and more) is natural.

Component testing checks individual units of code. Component testing is often called unit testing, but may also be called module testing or program testing. Developers write and execute unit tests to find and fix defects in their code as early as possible in the development process. This is critical in agile development environments, where short release cycles require fast test feedback. Unit tests are white-box tests because they are written with a knowledge of the code being checked.

A build that takes a long time to complete disrupts the CI process. To keep your testing efficient, approach your automated testing code as you would the application code itself. Regularly look for redundancies that can be eliminated, such as multiple tests that cover the same feature or data-driven tests that use repetitive data values. Techniques such as boundary value analysis and equivalence partitioning can help reduce your data-driven testing to just the essential cases.

Automated tests frequently bottleneck on network operations, or, more generally, use of external resources. When developers notice tests slowing, one of the remedies your refactoring can take is to mock those resources for fast results.

It’s essential that build tests complete as quickly as possible so that developers aren’t discouraged from committing frequently. To keep the process fast, trigger the minimum automated tests required to validate your build.

Due to their more complex nature, integration tests are usually slower than unit tests. Run your smoke and sanity tests first to rapidly identify a broken build before spending time on additional tests. If your team merges frequently, it may be more efficient to run integration tests only for daily builds rather than every merge.

Run a full regression only when necessary, such as in preparation for deployment to the production system. For example, Ranorex Studio supports the use of run configurations to run a subset of tests, such as smoke tests or tests just for features or modules of the application that have changed. Exclude from the build test set any low-priority regression test cases that haven’t found a defect in several test cycles.

To minimize the chance of test failures due to issues such as incorrect O/S version or missing services, test in an environment that is stable. Ideally, you will have an isolated test platform that is dedicated solely to testing. Your test environment should also be as identical as possible to the production environment, but this can be challenging. Realistically, it may be necessary to virtualize or mock certain dependencies such as third-party applications. In complex environments, a virtualization platform or solution such as Docker containers may be an efficient approach to replication of the production environment.

Speed is essential in a CI/CD environment, as two previous sections already mentioned. Quick return of results pays off enormously in making the most of developers’ flow. Save time by distributing your automated tests on a Selenium grid or running them in parallel on multiple physical or virtual servers. As mentioned earlier in this series, keep your automated tests as modular and independent of each other as possible so that you can test in parallel.

It takes a combination of automated testing approaches to confirm that your application is ready for deployment to the production environment. In addition to your automated unit and integration tests, include automated user interface tests to verify core functionality, check common user paths through the application end-to-end and validate complex workflows. Exploratory testing can uncover defects that automated tests miss.

After deploying the new build to the production environment, run your smoke tests in the production environment as a quick check to ensure a successful deployment.