Integrating Ranorex Automation in Jenkins CI Process

Posted by Tobias Walter on Tuesday, September 11th, 2012 at 9:49 am to Integration

In software engineering, continuous integration means the continuous application of quality control processes — small units of effort, applied frequently.

Hudson and Jenkins are open source tools written in Java for just such continuous integration.

This blog post will describe how to set up a CI job with Hudson/Jenkins that automatically builds and executes your Ranorex automation and automatically sends out the generated test reports for every committed change in a subversion repository.


Advantages of Continuous Integration

Continuous integration has many advantages:

  • When tests fail or bugs emerge, developers might revert the codebase to a bug-free state without wasting time for debugging
  • Developers detect and fix integration problems continuously – avoiding last-minute chaos at release dates
  • Early warning of broken/incompatible code
  • Early warning of conflicting changes
  • Immediate testing of all changes
  • Constant availability of a “current” build for testing, demo, or release purposes
  • Immediate feedback to developers on the quality, functionality, or system-wide impact of code they are writing
  • Frequent code check-in pushes developers to create modular, less complex code

Infrastructure

Continuous Integration Tool

A download link and an installation description for Hudson and Jenkins can be found at the following links:

In this blog post we are going to use Jenkins as CI tool, but using Hudson instead should not make much difference.

As Jenkins is normally started as a Windows service, it does not have sufficient rights to start UI-applications.

So, make sure that the Jenkins service is not started automatically as a service. To do so, open the “Services” tool (which is part of the “Administrative Tools” in the control panel), choose “Jenkins” service, stop the service, and set the “Startup type” to disabled:

Jenkins Properties

Jenkins can be started manually using following command (from the installation folder):

java -jar jenkins.war

After starting Jenkins, you can access the web interface on the machine you have installed it on using the following address:

http://localhost:8080/

You can configure your Jenkins installation according to your wishes navigating through the menu (“Manage Jenkins” -> “Configure System”):

Manage Jenkins

Note: It is necessary to have the Ranorex main components – and a valid Ranorex license – installed on each machine on which you are going to build and execute Ranorex code.

Source Code Management

As mentioned before, we are going to use a subversion repository as base of our continuous integration process.

In this sample we have two solutions placed in our repository: the application we are going to test and the testing application.

Repo Structure

To start the application under test from your test project, simply add a new “Run Application” action to your recorder action table which starts the application under test using a relative path to the repository root:

Add Run Application Action

Plugins

As we want to build our code for each committed change within our repository, we need a subversion as well as a MS Build plugin for Jenkins. To send a mail holding the test report after each build, we additionally need a mail plugin.

Install Plugins

To install the mentioned plugins, open the “Manage Plugins” section (“Manage Jenkins” -> “Configure System”), choose the following plugins from the list of available plugins and install them using the button on the bottom:

Configure Plugins

Next to installing the plugins, they have to be configured. Therefore open the “Manage System” section as described above and

Add New Job

Now, as the system is configured, we can add a new Jenkins job which will update the checked out files from a SVN repository, build both, the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send a mail with the report file attached.

First of all, create a new Job, choose “Build free-style software project” as job type and enter a name:

Add New Jenkins Job

Configure Source Code Management

Next, we have to check out the source of both the application under test and our test automation project. Therefore choose Subversion as source code management tool, enter the repository holding your application under test as well as your test automation project and choose “Use ‘svn update’ as much as possible” as check out strategy:

Configure SVN For Jenkins Job

With this configuration the application under test as well as the test automation project will be checked out and updated locally on the machine running Jenkins.

Add Build Steps

Now, as the source code management is configured, we can start with processing the updated files.
First of all, let’s add MSBuild steps for both projects:

Add MSBuild Step

Choose your configured MSBuild version and enter the path of the solution file relative to the repository root (which is the workspace folder of the Jenkins job) for both the automated and the automating project:

Add MSBuild Step

With adding these two build steps, the executables will be built automatically on the machine running Jenkins.

Now the newly built program should be tested.

This can be accomplished by adding a new “Windows batch command” starting the test suite executable (relative to the repository root which is the workspace folder root of the Jenkins job):

Add Execute Batch File Build Step

Added Test Execution Step

As you can see, some command line arguments are passed to the test suite executable.

In this sample the command line arguments “/zr”, which triggers the test suite executable to generate a zipped report file and “/zrf:.\Reports\Report-Build-%BUILD_NUMBER%.rxzlog” which defines the name and the location of the generated zipped report file, are used.

You can find a list of all available command line arguments in the section “Running Tests without Ranorex Studio” in our user guide.

The test suite executable returns “0” on success and “-1” on a failure. Based on this return value, Jenkins will mark the build as succeeded or failed.

Add Post-Build Action

After building and executing the application under test and the regarding test script, we will send an email which informs us about the success of the triggered build.

This email should include the zipped report file, mentioned before, as attachment.

Therefore add a new post-build action of type “Editable Email Notification”, choose the Report file location defined before as attachment, and add triggers for every job status you want to be informed about. In this sample an email will be sent if a job has failed or succeeded.

Added Mail Action

Run Job

After preforming these steps and saving your changes you can try out if everything works as expected by choosing “Build now”:

Build Now

After running the generated job you will see all finished builds within the build hierarchy with icons indicating the status of the individual builds.
You can view the zipped report files of all builds by opening them via the local workspace (“Workspace/Reports”):

Build History

As configured before, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as attachment.

Add Repository Hook

After performing the steps mentioned above, we are able to manually trigger a build. But as we are working with subversion, it would be beneficial to trigger the script for each commit.

Therefore you can add a server side repository hook, which automatically triggers Jenkins to start a new build for each change committed as described in the subversion plugin documentation.

Alternatively you can activate polling of the source code management system as build trigger in your Jenkins job configuration.

As shown in following picture, you can define the interval, the source code management will be invoked (e.g. 5 minutes after every full hour):

Added Build Trigger

Conclusion

Following the steps above you will be able to easily setup a continuous integration process performing the automated test of the application you develop.

After every commit a test automation is triggered that leads to automated feedback (success/failure) for the developer via mail with the Ranorex Report attached.

Please feel free to share your thoughts on this topic in the comments section.

Tobias Walter

Online Marketing and Content Specialist at Ranorex
Tobias Walter is Online Marketing and Content Specialist at Ranorex GmbH. Next to his marketing activities he also writes articles published in severals blogs and journals.

Latest posts by Tobias Walter (see all)

Share

Tags: , , , , , , , , , ,

31 comments

  1. Ben Simkins

    Can we have the same thing for TeamCity too, please? :)

  2. Andrew

    Same as Ben, a similar thing for TeamCity would be awesome for myself as well!

  3. Steve Stonebraker

    Why not just run the windows service as an administrative user?

    Additionally if you are using Tomcat on windows to run jenkins you just need to set the windows server to run as an administrative user.

  4. Ali Raza

    Does Ranorex produce some kind of x-unit compatible xml reports? If so, we can use Jenkins graphing/reporting features to get great metrics.

  5. twalter

    @Ben and Andrew
    We will have a look at TeamCity…

    @Steve
    Thanks for your remarks.

    @Ali
    Basically you can try execute your Ranorex Tests using the NUnit framework as described in following blog posts:
    http://www.ranorex.com/blog/combining-ranorex-3-0-and-visual-studio-2010-for-ui-unit-testing
    http://www.ranorex.com/blog/using-nunit-for-test-execution

  6. Ali

    Hi Twalter,
    Lets say I don’t want to refactor all my existing tests in Ranorex. Is there some command line way of exporting all test results to some XML file. I can then write a wrapper to pump those results into my reporting database

    Thanks

  7. twalter

    @Ali
    Basically the generated *.rxlog file is a XML file (and the rxzlog file includes the *.rxlog file).

  8. Ran

    Has any one tried integrating ranorex with Hudson????

  9. twalter

    @Ran
    Correct me if I’m wrong, but as I can see using Hudson works the same way as using Jenkins.

  10. Integrate Ranorex into Any Continuous Integration Process | Ranorex Blog

    [...] Integrating Ranorex Automation in Jenkins CI Process [...]

  11. Alin

    I tried to run my ranorex testcase with Jenkins, and it failed with following error message, however, it works when I ran “C:\Program Files (x86)\Jenkins\jobs\Test\workspace>.\test\test\bin\debug\test.exe /tc:TestCase /zr /zrf:.\Report\build-3.rxzlog”

    error message:
    [2012/12/05 16:17:42.911][Debug ][Logger]: Console logger starting.
    [2012/12/05 16:17:43.052][Info ][Test]: Test Case ‘TestCase’ started.
    [2012/12/05 16:17:43.052][Info ][Test]: Test Module ‘Recording1′ started.
    [2012/12/05 16:17:43.067][Info ][Keyboard]: Key sequence ‘{LWin down}r{LWin up}ca’.
    [2012/12/05 16:17:45.486][Info ][Keyboard]: Key sequence ‘lc{Return}’.
    [2012/12/05 16:17:52.476][Info ][Mouse]: Mouse Left Click item ‘Calculator.Calc.Button134′ at 9;8.
    [2012/12/05 16:17:52.538][Warn ][Java AWT/Swing]: Please make sure that you have correctly instrumented your Java installation or application. Otherwise Ranorex is not able to access UI objects inside your Java AWT/Swing application.
    The following website provides more information on this technology limitation:
    http://www.ranorex.com/support/user-guide-20/technology-instrumentation/testing-of-java-applications.html
    (This message is only shown once per report.)
    [2012/12/05 16:18:22.546][Error ][Module]: No element found for path ‘/form[@title='Calculator']‘ within 30s.The folder ‘testRepository.Calculator’ was not found.Failed to find item ‘testRepository.Calculator.Calc.Button134′.
    [2012/12/05 16:18:22.718][Failure][Test]: Test Module ‘Recording1′ completed with status ‘Failed’.
    [2012/12/05 16:18:22.718][Failure][Test]: Test Case ‘TestCase’ completed with status ‘Failed’.
    [2012/12/05 16:18:22.843][Debug ][Logger]: Console logger stopping.

    C:\Program Files (x86)\Jenkins\jobs\Test\workspace>exit -1

    key couldn’t be input even if testcase starts in log, so calculator didn’t show up.

  12. twalter

    @Alin
    Make sure that you have started Jenkins from command line and not as a service as described in the infrastructure section of the bog post.

  13. Alin

    Hi twalter,

    Thanks for your reply.
    it works well if start Jenkins from command line.

    and I have other question:
    the test suite name can be anything even if nothing by “test.exe /ts:”, it also can be ran.
    is that expected behavior?

    thanks again!

  14. twalter

    @Alin,
    the command line argument “testsuite|ts” can be used if you have more than one test suite files (*.rxtst) describing your test suite project. As the help text says, if nothing else is defined, the test suite file having the same name as the executable or the first test suite file in the same folder will be taken.
    May I ask what you are going to perform in detail?

  15. Bobby Dazzler

    Hello,

    I have successfully configured Jenkins to execute my Ranorex sutie and a Virtual Machine.

    It runs fine however when it comes to using the data connector it fails in locating the required file.

    [2013/04/25 17:20:35.894][Info ][Test]: Test Case ‘LocationsAndDepartments’ started.
    [2013/04/25 17:20:35.896][Info ][Test]: Test Module ‘Locations’ started.
    [2013/04/25 17:20:42.368][Info ][User]: Checking the Visility of the buttons…
    [2013/04/25 17:20:42.370][Success][Validation]: ‘{Button:Open Tree}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.372][Success][Validation]: ‘{Button:Close Tree}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.375][Success][Validation]: ‘{Button:Configure Company Details}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.377][Success][Validation]: ‘{Button:Modify}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.378][Success][Validation]: ‘{Button:Delete}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.380][Success][Validation]: ‘{Button:AddLocation}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.382][Success][Validation]: ‘{Button:AddDepartment}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.383][Success][Validation]: ‘{Button:AddUser}’ ‘Visible’ is being checked
    [2013/04/25 17:20:42.383][Info ][User]: Checking the Enabled state of the buttons…
    [2013/04/25 17:20:42.384][Success][Validation]: ‘{Button:Open Tree}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.385][Success][Validation]: ‘{Button:Close Tree}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.385][Success][Validation]: ‘{Button:Configure Company Details}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.386][Success][Validation]: ‘{Button:Modify}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.387][Success][Validation]: ‘{Button:Delete}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.388][Success][Validation]: ‘{Button:AddLocation}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.389][Success][Validation]: ‘{Button:AddDepartment}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.390][Success][Validation]: ‘{Button:AddUser}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:42.390][Info ][User]: Checking the edit box objects…
    [2013/04/25 17:20:45.674][Success][Validation]: ‘{Text:nameInput}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:45.675][Success][Validation]: ‘{Button:Cancel}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:45.677][Success][Validation]: ‘{Button:Create}’ ‘Enabled’ is being checked
    [2013/04/25 17:20:47.385][Success][Test]: Test Module ‘Locations’ completed with status ‘Success’.
    [2013/04/25 17:20:47.386][Info ][Test]: Test Module ‘LocationsDepsTreeBehaviour’ started.
    [2013/04/25 17:20:47.396][Error ][Module]: The Excel file with name ‘Traffic Live Automation Data.xlsx’ could not be found.
    [2013/04/25 17:20:47.422][Failure][Test]: Test Module ‘LocationsDepsTreeBehaviour’ completed with status ‘Failed’.
    [2013/04/25 17:20:47.422][Failure][Test]: Test Case ‘LocationsAndDepartments’ completed with status ‘Failed’.

    If I execute the Test Suite from the command line it locates it correctly. It is only when using this Batch Call from jenkins does it generate this error. Do I need to pass in a Global Parameter or is there any directove I need to pass to the batch command?

    The command is as follows:

    .\TrafficLIVE\bin\Release\Traffic_LIVE.exe

    The required file is located in the .\TrafficLIVE\bin\Release directory so I am at a bit of a loss.

    Any help would be appreciated.

    Thanks,

    Chris

  16. twalter

    @Chris

    try to change the working directory from your home directory to the “Release” directory before starting the test automation.
    The regarding command should look as follows:

    cd .\TrafficLIVE\bin\Release
    Traffic_LIVE.exe

    Regards,
    Tobias

  17. Bobby Dazzler

    @Tobias

    That was exactly my problem. Thank you very much for your swift reply.

    Regards,

    Chris

  18. EricM

    I cannot get Jenkins to execute the Ranorex build executable. The machine that the job is suppose to run on is a remote system that is up and running (user is disconnected by programs continue to run).

    The job never runs, unless I remote desktop to the system and keep the remote destop session open, which I should not have to do. Has anyone had this issue?

    Thanks,
    Eric

  19. twalter

    @EricM

    as you are going to automate a graphical user interface it is necessary to have a screen where this UI can be executed at.
    By closing the remote session the display will be locked and therefore the automation will fail.
    Please have a look at the following chapter of our user guide for further details:
    http://www.ranorex.com/support/user-guide-20/remotely-working-with-ranorex.html

    Regards,
    Tobias

  20. TestXYZ

    Hi twalter,
    I have started Build-Job (MS Build) in Jenkins, as described in the post but I got the error about Hudson in the console output:

    … (MSBuild succeeded)

    Time Elapsed 00:00:00.72
    [workspace] $ sh -xe C:\Windows\TEMP\hudson8696041774381560790.sh
    The system cannot find the file specified
    FATAL: Befehlsausführung fehlgeschlagen
    java.io.IOException: Cannot run program “sh” (in directory “C:\Program Files\Jenkins\jobs\Systemtest-BASIS_MEDICAL_FREMDKATALOG\workspace”): CreateProcess error=2, Das System kann die angegebene Datei nicht finden
    at java.lang.ProcessBuilder.start(Unknown Source)
    at hudson.Proc$LocalProc.(Proc.java:244)
    at hudson.Proc$LocalProc.(Proc.java:216)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:773)
    at hudson.Launcher$ProcStarter.start(Launcher.java:353)
    at hudson.Launcher$ProcStarter.join(Launcher.java:360)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
    at hudson.model.Build$BuildExecution.build(Build.java:199)
    at hudson.model.Build$BuildExecution.doRun(Build.java:160)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:567)
    at hudson.model.Run.execute(Run.java:1603)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:246)
    Caused by: java.io.IOException: CreateProcess error=2, Das System kann die angegebene Datei nicht finden
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    … 17 more
    Build step ‘Shell ausführen’ marked build as failure
    No emails were triggered.
    Finished: FAILURE

    What’s wrong? Thank you very much! Best Regards

  21. TestXYZ

    Hello twalter,

    about the above problem I think I should rewrite the Shellscript, only use “Windows Batch” then it works fine. The only problem is: if I start Jenkins from command line “java –jar Jenkins.war” then I can not login and manage Jenkins (my account is valid and has admin role), Error: http://magv-mccqa01:8080/j_acegi_security_check ((HTTP 404 not found)

    But if I start Jenkins as a windows service then it does not have sufficient rights to start Ranorex UI-tests. Maybe it is a problem in Jenkins on Windows 7? Many Thanks, Best Regards

  22. Tobias Walter

    @TestXYZ: You have to make sure to have a desktop on which the UI tests can be executed. That can be established by starting jenins form command line. Basically I did not have any problem with logging in on jenkins web ui when starting Jenkins from command line. Looks like you have some kind of configuration problems. Try to check your config or reconfigure jenkins.

    Regards, Tobias

  23. TestXYZ

    Hello Tobias,
    Thanks for your answer. Yes that is config problem in Jenkins. It maybe relate to https://issues.jenkins-ci.org/browse/JENKINS-3761. But I have uninstalled all and remove manually the configuration files from the folder C:\{user}\.jenkins\*.*. Then I have installed Jenkins again but without any security settings. Now it works fine so far so on :-) Best Regards

  24. Tobias Walter

    @TestXYZ: Good to hear, that your system is now running as expected…
    Regards

  25. Bengt Johansson

    Hi Tobias, I really liked this presentation especially the Jenkins overview picture. I wonder if I may use it in a presentation I will give to students at the Lund Institute of Technology?

  26. Tobias Walter

    Hi Bengt,
    Sure, you can use the illustration for your presentation.

  27. Darius

    Hello,

    I want to test a Android App. How can I select the built apk file to be deployed and instrumented in the emulator that I created on Jenkins?

    Thank you!

  28. Tobias Walter

    @Darius
    You can deploy the apk file via command line as described in the chapter “Running Instrumentation Wizard from the Command Line”. Additionally the apk file can be instrumented on the same way.

  29. Tobias Walter

    If you are having trouble getting the agent installed as a Windows service (as you won’t be able to run UI tests), have a look at the article on http://jenkins-ci.org/ describing how to start the agent via Windows Scheduler:
    https://wiki.jenkins-ci.org/display/JENKINS/Launch+Java+Web+Start+slave+agent+via+Windows+Scheduler

  30. Make your Ranorex Report comprehensible for Jenkins | Ranorex Blog

    [...] blog post “Integrating Ranorex Automation in Jenkins CI Process” is introducing the entire CI topic and gives a step by step introduction on how to integrate [...]

  31. Darius

    Hello Tobias,

    I managed to run the Ranorex-Jenkins-Androdi integration. There is still one issue. If I want to run the tests from jenkins on a device, it says that “the selected devices is not in the devices list”. I have some batch files which uninstalls, installs the app and runs the tests. If I run the batch files manually, it works fine. If I want to run the batch files from Jenkins, I receive the error above. The workaround that I have is to create another solution which only clicks the batch files. I guess that Jenkins does not have permission to read the list of devices from Ranorex.
    Did you manage to get Ranorex-Jenkins working with Android devices? If yes, can you please write me some instructions?

    Thank you!
    Darius

Leave a Reply