In software engineering, continuous integration means the continuous application of quality control processes — small units of effort, applied frequently.
In this blog we’ll show you how to set up a CI job with Hudson/Jenkins that automatically builds and executes your Ranorex automation as well as automatically sends out the generated test reports for every committed change in a Subversion repository.
- Advantages of Continuous Integration Testing
- Add New Job
- Run Job
- Add Repository Hook
Advantages of Continuous Integration Testing
Continuous integration has many advantages:
- When tests fail or bugs emerge, developers can revert the codebase to a bug-free state without wasting time for debugging
- Developers detect and fix integration problems continuously – and thus avoid last-minute chaos at release dates
- Early warning of broken/incompatible code
- Early warning of conflicting changes
- Immediate testing of all changes
- Constant availability of a “current” build for testing, demo, or release purposes
- Immediate feedback to developers on the quality, functionality, or system-wide impact of their written code
- Frequent code check-ins push developers to create modular, less complex code
Continuous Integration Tool
You can find a download link and installation description for Hudson and Jenkins here:
In this blog post we are going to use Jenkins as CI tool. There shouldn’t be much of a difference when using Hudson.
As Jenkins or the nodes executing the CI jobs normally are started as Windows services, they do not have sufficient rights to start UI-applications.
Please make sure that Jenkins as master and its slave nodes, where the Ranorex automation should be triggered, are not started as a service.
For the Jenkins master, open the “Services” tool (which is part of the “Administrative Tools” in the control panel), choose “Jenkins” service, stop the service, and set the “Startup type” to disabled:
Use the following command to start Jenkins manually from the installation folder:
java -jar jenkins.war
After starting Jenkins, use this address to access the web interface:
To configure your Jenkins server, navigate to the Jenkins menu and select “Manage Jenkins” -> “Configure System”:
Note: It is necessary to have the Ranorex main components – and a valid Ranorex license – installed on each machine you want to build and execute Ranorex code.
Source Code Management
As mentioned before, we are going to use a Subversion repository as base of our continuous integration process.
In this sample, we have two solutions in our repository: the application under test and as the automated Ranorex tests.
To start the application under test from your test project, simply add a new “Run Application” action to your action table in Ranorex Studio, which starts the application under test, using a relative path to the repository root:
As we want to build our code for each committed change within our SVN repository, we need a Subversion as well as a MS Build plugin for Jenkins. An additional mail plugin will make sure that a mail is sent with each build.
Open the “Manage Plugins” section (“Manage Jenkins” -> “Manage Plugins”), choose the following plugins from the list of available plugins and install them if they are not installed already:
- MSBuild Plugin
- Email Extension Plugin
- Subversion Plugin
- Ranorex Test Execution
The installed plugins also need to be configured. To do so
- open the “Configure System” and configure the “Extended E-Mail Notification” plugin. To do so, set the recipients and alter the subject and content (adding the environment variable $BUILD_LOG to the content will add the whole console output of the build and the test to the sent mail),
- configure the “E-mail Notification” plugin by setting the SMTP server.
- and navigate to “Global Tool Configuraiton” and configure your “MSBuild” plugin by choosing the “msbuild.exe” installed on your machine.
Add New Job
Now, as the system is configured, we can add a new Jenkins job, which will update the checked out files from a SVN repository, build both the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send a mail with the report file attached.
Start by creating a new item. Choose “Build free-style software project” as job type and enter a job name:
Configure Source Code Management
Next, we have to check out the source of both the application under test and our test automation project. Start with choosing Subversion as source code management tool. Then, enter the repository holding your application under test as well as your test automation project. Finally, choose “Use ‘svn update’ as much as possible” as check out strategy:
With this configuration, the application under test as well as the test automation project will be checked out and updated locally.
Add Build Steps
Now, as the source code management is configured, we can start with processing the updated files.
First of all, let’s add MSBuild steps for both projects:
Choose your configured MSBuild version and enter the path of the solution file relative to the repository root (which is the workspace folder of the Jenkins job) for both the automated and the automating project:
Add “Run a Ranorex Test Suite” Step
By adding these two build steps, the executables will be automatically built. Now the newly built application should be tested.
How to set up the „Run a Ranorex test suite” build step
- Ranorex test suite file: Enter the path to the test suite file (*.rxtst) located in the output folder of your solution.
- Ranorex run configuration: Enter the exact name of the run configuration you want to use. By default, the run configuration currently selected in the test suite is used. If you want to create or edit run configurations, please use Ranorex Studio or the Ranorex Test Suite Runner.
- Ranorex report directory: Specify the directory that your report (and accordingly the JUnit report file) will be saved to. If you don’t specify a path, the directory where your test executable is located will be used.
- Ranorex report file name: Specify the file name of the generated report. By default, the file name specified in the test suite settings is used.
- JUnit-compatible report: If checked, Ranorex will create both a JUnit-compatible report and a Ranorex report.
- Compressed copy of Ranorex report: Compresses the report and the associated files into a single archive with the extension .rxzlog. The following additional input fields will appear when this option is enabled:
- Compressed report directory: Allows you to specify the directory that your compressed report will be saved to. If you don’t specify a path, the directory where your test executable is located will be used.
- Compressed report file: Allows you to specify the file name of the compressed report file. By default, the file name specified in the test suite settings is used.
- Global parameters: Allows you to create or override values for global parameters set in the test suite. Enter parameters according to the following pattern: “ParameterName=Value”. Separate parameters with semicolons or newlines.
- Command line arguments: Allows you to add additional test suite command line arguments. Please refer to our user guide to get a list of all available command line arguments.
The test suite executable returns “0” on success and “-1” on failure. Based on this return value, Jenkins will mark the build as successful or failed.
Add Post-Build Action
After executing the automated tests, we will publish a JUnit test result report to the Jenkins build. For this reason, we have created a JUnit compatible copy of the report file, by checking the corresponding option. Now, we can add a “Publish Junit test result report” post-build action to the build job and define the test report xml, representing your JUnit report file.
Additionally, we will send an email which informs us about the success of the triggered build.
This email should include the zipped report file, mentioned before, as attachment.
To do so, add the new post-build action “Editable Email Notification”, choose the report file location defined before as attachment, and add triggers for each job status you want to be informed about. In this sample, an email will be sent if a job has failed or succeeded.
Once you’ve completed these steps and saved your changes, check if everything works as expected by clicking “Build now”:
After running the generated job, you will see all finished builds within the build history. Icons indicate the status of the individual builds. If you click on a specific build, you will see the detailed JUnit test results. You can view the zipped report files of all builds by opening them in the local workspace (“Workspace/Reports”):
As configured before, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as attachment.
Add Repository Hook
Now we can manually trigger a build. As we are working with Subversion, it would be beneficial to trigger the script for each commit.
To do so, you can add a server side repository hook, which automatically triggers Jenkins to start a new build for each change committed, as described in the subversion plugin documentation.
Alternatively, you can activate polling of the source code management system as build trigger in your Jenkins job configuration.
As shown in following picture, you can define the interval, after which the source code management will be invoked (e.g. 5 minutes after every full hour):
Following the steps above you will be able to easily setup a continuous integration process performing the automated test of the application you develop. Each commit will now trigger an automated test run. Once the test run has finished, you’ll instantly receive a mail with the Ranorex test report.
Note: This blog was originally published in July 2012 and has been revised to reflect recent technical developments.