In software engineering, continuous integration (CI) means the continuous application of quality control processes — small units of effort, applied frequently.
In this blog, we’ll show you how to set up a CI job with Hudson/Jenkins that automatically builds and executes Ranorex automated tests and generates test reports for every change committed to a Subversion repository.
- Advantages of Continuous Integration Testing
- Add a New Job
- Run a Job
- Add a Repository Hook
Advantages of Continuous Integration Testing
Continuous integration has many advantages. When tests fail or bugs emerge, developers can revert the codebase to a bug-free state without wasting time debugging. Developers can detect and fix integration problems continuously—and thus avoid last-minute chaos at release dates. In addition, CI testing provides:
- Early warning of broken/incompatible code
- Early warning of conflicting changes
- Immediate testing of all changes
- Constant availability of a “current” build for testing, demo, or release purposes
- Immediate feedback to developers on the quality, functionality, or system-wide impact of their written code
- Frequent code check-ins push developers to create modular, less complex code
Continuous Integration Tools
Below are download links and installation description for Hudson and Jenkins:
This blog post will use Jenkins as the CI tool. Differences between Jenkins and Hudson are minimal. Note that because Jenkins or the nodes executing the CI jobs normally are started as Windows services, they do not have sufficient rights to start UI-applications. Ensure that Jenkins is not started as a service on the master or on the slave nodes where the Ranorex automation should be triggered.
For the Jenkins master, open the Services tool (which is part of the “Administrative Tools” in the control panel), choose the Jenkins service, stop the service, and set the Startup type to Disabled, as shown below.
Use the following command to start Jenkins manually from the installation folder:
java -jar jenkins.war
After starting Jenkins, use this address to access the web interface: http://localhost:8080/
To configure your Jenkins server, navigate to the Jenkins menu and select Manage Jenkins > Configure System:
Note: It is necessary to have the Ranorex main components—and a valid Ranorex license—installed on each machine where you want to build and execute Ranorex code.
Source Code Management
As mentioned before, this example uses a Subversion repository as the base for the continuous integration process. In this sample, we have two solutions in the repository: the application under test and the automated Ranorex tests.
To start the application under test from your test project, simply add a new Run Application action to your action table in Ranorex Studio, which starts the application under test, using a relative path to the repository root:
Since we want to build the code for each committed change within the SVN repository, we need a Subversion and an MS Build plugin for Jenkins. An additional mail plugin will ensure that an email is sent with each build.
From the Jenkins menu, choose Manage Jenkins > Manage Plugins. Select the following plugins from the list of available plugins and install them if they are not already installed:
- MSBuild Plugin
- Email Extension Plugin
- Subversion Plugin
- Ranorex Test Execution
The installed plugins also need to be configured, following the steps below:
- From the Manage Jenkins menu, choose Configure System. Configure the Extended E-Mail Notification plugin with the recipient(s). Set the default subject and default content as shown below. Including the environment variable $BUILD_LOG with the content will add the whole console output of the build and the test to the generated email.
- Configure the E-mail Notification plugin by setting the SMTP server.
- Navigate to Global Tool Configuration and configure your MSBuild plugin by choosing the msbuild.exe installed on your machine.
Add a New Job
Now that the system is configured, you can add a new Jenkins job. This job will update the checked-out files from an SVN repository, build both the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send an email with the report file attached.
Start by creating a new item. Choose Build free-style software project as job type and enter a job name:
Configure Source Code Management
The next step is to check out the source of both the application under test and the test automation project. Start with choosing Subversion as the source code management tool. Then, enter the repository containing your application under test as well as your test automation project. Finally, select Use ‘svn update’ as much as possible as the check-out strategy:
With this configuration, the application under test as well as the test automation project will be checked out and updated locally.
Add Build Steps
Now that the source code management is configured, you can start with processing the updated files. First, add MSBuild steps for both projects:
Choose the configured MSBuild version and enter the path of the solution file relative to the repository root (the workspace folder of the Jenkins job), for both the automated and the automating project:
Add “Run a Ranorex Test Suite” Step
By adding these two build steps, the executables will be built automatically. Now the newly-built application should be tested. This can be accomplished by adding a new “Run a Ranorex test suite” build step that starts the test suite executable.
To see how this works, watch the following short video or follow the steps below:
How to configure the “Run a Ranorex test suite” build step
- Ranorex test suite file: Enter the path to the test suite file (*.rxtst) located in the output folder of your solution.
- Ranorex run configuration: Enter the exact name of the run configuration you want to use. By default, the run configuration currently selected in the test suite is used. If you want to create or edit run configurations, please use Ranorex Studio or the Ranorex Test Suite Runner.
- Ranorex report directory: Specify the directory where your report and corresponding JUnit report file will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
- Ranorex report file name: Specify the file name of the generated report. By default, the file name specified in the test suite settings is used.
- JUnit-compatible report: If checked, Ranorex will create both a JUnit-compatible report and a Ranorex report.
- Compressed copy of Ranorex report: Compresses the report and the associated files into a single archive with the extension .rxzlog. The following additional input fields will appear when this option is enabled:
- Compressed report directory: Set the directory where your compressed report will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
- Compressed report file: Set the file name of the compressed report file. By default, the file name specified in the test suite settings is used.
- Global parameters: Create or override values for global parameters set in the test suite. Enter parameters according to the following pattern: “ParameterName=Value”. Separate parameters with semicolons or newlines.
- Command line arguments: Enter additional test suite command line arguments. Please refer to our user guide to get a list of all available command line arguments.
The test suite executable returns “0” on success and “-1” on failure. Based on this return value, Jenkins will mark the build as successful or failed.
Add Post-Build Action
If you selected the “JUnit-compatible report” option in the previous step, now you can publish a JUnit test result report to the Jenkins build after the automated tests complete. In this step, you will add a “Publish Junit test result report” post-build action to the build job and define the test report xml, representing your JUnit report file.
In addition, you can send an email regarding the success of the triggered build. This email should include the zipped report file, mentioned before, as an attachment. To do so, add the new post-build action “Editable Email Notification”, choose the report file location defined before as attachment, and add triggers for each job status you want to be informed about. In this sample, an email will be sent if a job has failed or succeeded.
Run a Job
Once you have completed these steps and saved your changes, verify that everything works as expected by clicking Build now:
After running the generated job, you should see all finished builds within the build history. Icons indicate the status of the individual builds. If you click on a specific build, you will see the detailed JUnit test results. View the zipped report files of all builds by opening them in the local workspace (“Workspace/Reports”):
As configured in the previous steps, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as an attachment.
Add a Repository Hook
Now we can manually trigger a build. Because we are working with Subversion in this example, it would be beneficial to trigger the script for each commit. To do so, add a server-side repository hook, which automatically triggers Jenkins to start a new build for each change committed, as described in the subversion plugin documentation.
Alternatively, you can activate polling of the source code management system as a build trigger in your Jenkins job configuration. As shown in the following screenshot, you can define the interval, after which the source code management will be invoked (e.g. 5 minutes after every full hour):
Following the steps above, you will be able to set up a continuous integration process that includes Ranorex automated tests for the application under test. Each commit will trigger an automated test run. Once the test run has finished, you’ll instantly receive an email with the Ranorex test report.
Note: This blog was originally published in July 2012 and has been revised to reflect recent technical developments.