Rapid application development practices such as Continuous Integration (CI), Continuous Deployment (CD), and DevOps have a common goal: small, frequent releases of high-quality, “working” software. Whether your development cycle is measured in weeks or days, integrated automated tests are essential to maintaining the pace of development.
Automated tests in a CI pipeline
The image below shows a typical CI pipeline. A developer checks out code from the shared repository in the version control system, such as Git, TFS or Subversion. Once code changes are complete, the developer commits the change back to the version control system, triggering a CI job. The CI server builds the application under test and triggers automated tests to verify whether the new code results in a good, “green” build. The results of testing are reported back to the entire team for a decision regarding deployment of the application. In a CD environment, the application is deployed automatically to the production environment.
A continuous integration pipeline
Continuous integration with automated testing offers several benefits to organizations, including the following:
- Developers get fast feedback on the quality, functionality, or system-wide impact of their code changes, when defects are easier and less expensive to fix.
- Frequent integration of small changes reduces the volume of merge conflicts that can occur when several developers are working on the same application code, and makes merge conflicts easier to resolve when they do happen.
- Everyone on the team has a clear understanding of the status of the build.
- A current “good build” of the application is always available for testing, demonstration, or release.
Recommendations for automated testing in a CI pipeline
The recommendations below focus on test automation in a CI pipeline, some of which overlap the best practices for the CI process itself. Read more about best practices for a CI process in the Wikipedia article on Continuous Integration.
Don't rely solely on unit tests
Unit testing in an individual developer’s local environment doesn’t tell you enough about how that code will work once it is introduced to the production application. Integration of new or revised code may cause a build to fail for several reasons. For example, changes made by another developer may conflict with the new code, or there may be differences between the developer’s local environment and the production environment. Therefore, it’s important to run integration tests, regression tests and high-priority functional UI tests as part of the build verification process.
Make your build self-testing
Component testing checks individual units of code. Component testing is often called unit testing, but may also be called module testing or program testing. Developers write and execute unit tests to find and fix defects in their code as early as possible in the development process. This is critical in agile development environments, where short release cycles require fast test feedback. Unit tests are white-box tests because they are written with a knowledge of the code being checked.
Refactor your automated tests
A build that takes a long time to complete disrupts the CI process. To keep your testing efficient, approach your automated testing code as you would the application code itself. Regularly look for redundancies that can be eliminated, such as multiple tests that cover the same feature or data-driven tests that use repetitive data values. Techniques such as boundary value analysis and equivalence partitioning can help reduce your data-driven testing to just the essential cases.
Keep your build fast
It’s essential that build tests complete as quickly as possible so that developers aren’t discouraged from committing frequently. To keep the process fast, trigger the minimum automated tests required to validate your build. Due to their more complex nature, integration tests are usually slower than unit tests. Run your smoke and sanity tests first to rapidly identify a broken build before spending time on additional tests. If your team merges frequently, it may be more efficient to run integration tests only for daily builds rather than every merge. Run a full regression only when necessary, such as in preparation for deployment to the production system. For example, Ranorex Studio supports the use of run configurations to run a subset of tests, such as smoke tests or tests just for features or modules of the application that have changed. Exclude from the build test set any low-priority regression test cases that haven’t found a defect in several test cycles.
Use source control for your automated tests
Maintain your automated tests in the same repository as your code. This will make it easier to match the correct version of a test to the version of the source code. Ranorex Studio integrates with popular solutions for source control including Git, Microsoft Team Foundation Server, and Subversion.
Test in the right environment
To minimize the chance of test failures due to issues such as incorrect O/S version or missing services, test in an environment that is stable. Ideally, you will have an isolated test platform that is dedicated solely to testing. Your test environment should also be as identical as possible to the production environment, but this can be challenging. Realistically, it may be necessary to virtualize certain dependencies such as third-party applications. In complex environments, a virtualization platform or solution such as Docker containers may be an efficient approach to replicating the production environment.
Test in parallel
Speed is essential in a CI/CD environment. Save time by distributing your automated tests on a Selenium grid or running them in parallel on multiple physical or virtual servers. As mentioned earlier in this series, keep your automated tests as modular and independent of each other as possible so that you can test in parallel.
Include functional UI and exploratory testing
It takes a combination of automated testing approaches to confirm that your application is ready for deployment to the production environment. In addition to your automated unit and integration tests, include automated user interface tests to verify core functionality, check common user paths through the application end-to-end and validate complex workflows. Exploratory testing can uncover defects that automated tests miss.
Verify your deployment
After deploying the new build to the production environment, run your smoke tests in the production environment as a quick check to ensure a successful deployment.
To learn more about how to integrate Ranorex Studio tests in your CI pipeline, read our blog article “Integrate Automated Testing into Jenkins.” While this article focuses on Jenkins, Ranorex tests can be triggered from any CI server.