Integrate Automated Testing Into Jenkins

Mar 7, 2018 | Integration, Product Insights

Jenkings Integration


In software engineering, continuous integration (CI) means the continuous application of quality control processes — small units of effort, applied frequently.

In this blog, we’ll show you how to set up a CI job with Hudson/Jenkins that automatically builds and executes Ranorex automated tests and generates test reports for every change committed to a Subversion repository.

Watch our on-demand webinar

Watch our on-demand webinar

Mastering Jenkins Integration: See how easy it is to integrate Ranorex automated tests into your Jenkins build management and deployment process.


Advantages of Continuous Integration Testing

Continuous integration has many advantages. When tests fail or bugs emerge, developers can revert the codebase to a bug-free state without wasting time debugging. Developers can detect and fix integration problems continuously—and thus avoid last-minute chaos at release dates. In addition, CI testing provides:

  • Early warning of broken/incompatible code
  • Early warning of conflicting changes
  • Immediate testing of all changes
  • Constant availability of a “current” build for testing, demo, or release purposes
  • Immediate feedback to developers on the quality, functionality, or system-wide impact of their written code
  • Frequent code check-ins push developers to create modular, less complex code

Return to top


Continuous Integration Tools

Below are download links and installation description for Hudson and Jenkins:

This blog post will use Jenkins as the CI tool. Differences between Jenkins and Hudson are minimal. Note that because Jenkins or the nodes executing the CI jobs normally are started as Windows services, they do not have sufficient rights to start UI-applications. Ensure that Jenkins is not started as a service on the master or on the slave nodes where the Ranorex automation should be triggered.

For the Jenkins master, open the Services tool (which is part of the “Administrative Tools” in the control panel), choose the Jenkins service, stop the service, and set the Startup type to Disabled, as shown below.

disable start as service

Use the following command to start Jenkins manually from the installation folder:

java -jar jenkins.war

manually start jenkins

After starting Jenkins, use this address to access the web interface: http://localhost:8080/

To configure your Jenkins server, navigate to the Jenkins menu and select Manage Jenkins > Configure System:

Configure System

Note: It is necessary to have the Ranorex main components—and a valid Ranorex license—installed on each machine where you want to build and execute Ranorex code.

Return to top

Source Code Management

As mentioned before, this example uses a Subversion repository as the base for the continuous integration process. In this sample, we have two solutions in the repository: the application under test and the automated Ranorex tests.


To start the application under test from your test project, simply add a new Run Application action to your action table in Ranorex Studio, which starts the application under test, using a relative path to the repository root:

Run Application Action

Return to top


Since we want to build the code for each committed change within the SVN repository, we need a Subversion and an MS Build plugin for Jenkins. An additional mail plugin will ensure that an email is sent with each build.

Install Plugins

From the Jenkins menu, choose Manage Jenkins > Manage Plugins. Select the following plugins from the list of available plugins and install them if they are not already installed:

Configure Plugins

The installed plugins also need to be configured, following the steps below:

  • From the Manage Jenkins menu, choose Configure System. Configure the Extended E-Mail Notification plugin with the recipient(s). Set the default subject and default content as shown below. Including the environment variable $BUILD_LOG with the content will add the whole console output of the build and the test to the generated email.

Configure Emails

  • Configure the E-mail Notification plugin by setting the SMTP server.
  • Navigate to Global Tool Configuration and configure your MSBuild plugin by choosing the msbuild.exe installed on your machine.

Configure MSBuild

Return to top

Add a New Job

Now that the system is configured, you can add a new Jenkins job. This job will update the checked-out files from an SVN repository, build both the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send an email with the report file attached.

Start by creating a new item. Choose Build free-style software project as job type and enter a job name:

Add New Item

Return to top

Configure Source Code Management

The next step is to check out the source of both the application under test and the test automation project. Start with choosing Subversion as the source code management tool. Then, enter the repository containing your application under test as well as your test automation project. Finally, select Use ‘svn update’ as much as possible as the check-out strategy:

Configure SVN

With this configuration, the application under test as well as the test automation project will be checked out and updated locally.

Return to top

Add Build Steps

Now that the source code management is configured, you can start with processing the updated files. First, add MSBuild steps for both projects:

Add MSBuild Buildstep

Choose the configured MSBuild version and enter the path of the solution file relative to the repository root (the workspace folder of the Jenkins job), for both the automated and the automating project:

Added MSBuild Buildsteps

Return to top

Add “Run a Ranorex Test Suite” Step

By adding these two build steps, the executables will be built automatically. Now the newly-built application should be tested. This can be accomplished by adding a new “Run a Ranorex test suite” build step that starts the test suite executable.

To see how this works, watch the following short video or follow the steps below:



How to configure the “Run a Ranorex test suite” build step

  • Ranorex test suite file: Enter the path to the test suite file (*.rxtst) located in the output folder of your solution.
  • Ranorex run configuration: Enter the exact name of the run configuration you want to use. By default, the run configuration currently selected in the test suite is used. If you want to create or edit run configurations, please use Ranorex Studio or the Ranorex Test Suite Runner.
  • Ranorex report directory: Specify the directory where your report and corresponding JUnit report file will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
  • Ranorex report file name: Specify the file name of the generated report. By default, the file name specified in the test suite settings is used.
  • JUnit-compatible report: If checked, Ranorex will create both a JUnit-compatible report and a Ranorex report.
  • Compressed copy of Ranorex report: Compresses the report and the associated files into a single archive with the extension .rxzlog. The following additional input fields will appear when this option is enabled:
    • Compressed report directory: Set the directory where your compressed report will be saved. If you don’t specify a path, the directory where your test executable is located will be used.
    • Compressed report file: Set the file name of the compressed report file. By default, the file name specified in the test suite settings is used.
  • Global parameters: Create or override values for global parameters set in the test suite. Enter parameters according to the following pattern: “ParameterName=Value”. Separate parameters with semicolons or newlines.
  • Command line arguments: Enter additional test suite command line arguments. Please refer to our user guide to get a list of all available command line arguments.

The test suite executable returns “0” on success and “-1” on failure. Based on this return value, Jenkins will mark the build as successful or failed.

Return to top

Add Post-Build Action

If you selected the “JUnit-compatible report” option in the previous step, now you can publish a JUnit test result report to the Jenkins build after the automated tests complete. In this step, you will add a “Publish Junit test result report” post-build action to the build job and define the test report xml, representing your JUnit report file.

Added JUnit Action

In addition, you can send an email regarding the success of the triggered build. This email should include the zipped report file, mentioned before, as an attachment. To do so, add the new post-build action “Editable Email Notification”, choose the report file location defined before as attachment, and add triggers for each job status you want to be informed about. In this sample, an email will be sent if a job has failed or succeeded.

Added Mail Action

Return to top

Run a Job

Once you have completed these steps and saved your changes, verify that everything works as expected by clicking Build now:

Build Now

After running the generated job, you should see all finished builds within the build history. Icons indicate the status of the individual builds. If you click on a specific build, you will see the detailed JUnit test results. View the zipped report files of all builds by opening them in the local workspace (“Workspace/Reports”):

Build History

As configured in the previous steps, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as an attachment.

Return to top

Add a Repository Hook

Now we can manually trigger a build. Because we are working with Subversion in this example, it would be beneficial to trigger the script for each commit. To do so, add a server-side repository hook, which automatically triggers Jenkins to start a new build for each change committed, as described in the Subversion plugin documentation.

Alternatively, you can activate polling of the source code management system as a build trigger in your Jenkins job configuration. As shown in the following screenshot, you can define the interval, after which the source code management will be invoked (e.g. 5 minutes after every full hour):

Added Build Trigger

Return to top


Following the steps above, you will be able to set up a continuous integration process that includes Ranorex automated tests for the application under test. Each commit will trigger an automated test run. Once the test run has finished, you’ll instantly receive an email with the Ranorex test report.

Note: This blog was originally published in July 2012 and has been revised to reflect recent technical developments.

Manage Your Source Code Securely

Hosted and on-prem solutions for Git, SVN and Perforce

All-in-one Test Automation

Cross-Technology | Cross-Device | Cross-Platform

Related Posts:

How Does Ranorex Rank on Review Sites?

How Does Ranorex Rank on Review Sites?

If you're part of a software team, you know how challenging and rewarding user interface (UI) testing can be. After all, it's a time-consuming process that involves technical knowledge and experience. But the feeling of a successful UI test—when your application runs...

Exploring 5 Types of Usability Testing

Exploring 5 Types of Usability Testing

  Usability testing gives you valuable insights into how real users interact with your product. You can use the reactions and feedback to refine your product to ensure it’s effective and enhances the user’s enjoyment. We’ll explore different types of usability...

Secure Your Code, Harden Your App, Automate Its Testing

Secure Your Code, Harden Your App, Automate Its Testing

With DevOps practices more popular than ever in software engineering, there has been a push to integrate security, optimization, and frequent testing into the development process. However, as the baseline for what's considered good software evolves, so does the need...