In software engineering, continuous integration means the continuous application of quality control processes — small units of effort, applied frequently.
Hudson and Jenkins are open source tools written in Java for just such continuous integration.
This blog post will describe how to set up a CI job with Hudson/Jenkins that automatically builds and executes your Ranorex automation and automatically sends out the generated test reports for every committed change in a subversion repository.
- Advantages of Continuous Integration
- Infrastructure
- Plugins
- Add New Job
- Run Job
- Add Repository Hook
- Conclusion
Advantages of Continuous Integration
Continuous integration has many advantages:
- When tests fail or bugs emerge, developers might revert the codebase to a bug-free state without wasting time for debugging
- Developers detect and fix integration problems continuously – avoiding last-minute chaos at release dates
- Early warning of broken/incompatible code
- Early warning of conflicting changes
- Immediate testing of all changes
- Constant availability of a “current” build for testing, demo, or release purposes
- Immediate feedback to developers on the quality, functionality, or system-wide impact of code they are writing
- Frequent code check-in pushes developers to create modular, less complex code
Infrastructure
Continuous Integration Tool
A download link and an installation description for Hudson and Jenkins can be found at the following links:
In this blog post we are going to use Jenkins as CI tool, but using Hudson instead should not make much difference.
As Jenkins is normally started as a Windows service, it does not have sufficient rights to start UI-applications.
So, make sure that the Jenkins service is not started automatically as a service. To do so, open the “Services” tool (which is part of the “Administrative Tools” in the control panel), choose “Jenkins” service, stop the service, and set the “Startup type” to disabled:
Jenkins can be started manually using following command (from the installation folder):
java -jar jenkins.war
After starting Jenkins, you can access the web interface on the machine you have installed it on using the following address:
You can configure your Jenkins installation according to your wishes navigating through the menu (“Manage Jenkins” -> “Configure System”):
Note: It is necessary to have the Ranorex main components – and a valid Ranorex license – installed on each machine on which you are going to build and execute Ranorex code.
Source Code Management
As mentioned before, we are going to use a subversion repository as base of our continuous integration process.
In this sample we have two solutions placed in our repository: the application we are going to test and the testing application.
To start the application under test from your test project, simply add a new “Run Application” action to your recorder action table which starts the application under test using a relative path to the repository root:
Plugins
As we want to build our code for each committed change within our repository, we need a subversion as well as a MS Build plugin for Jenkins. To send a mail holding the test report after each build, we additionally need a mail plugin.
Install Plugins
To install the mentioned plugins, open the “Manage Plugins” section (“Manage Jenkins” -> “Configure System”), choose the following plugins from the list of available plugins and install them using the button on the bottom:
- MSBuild Plugin
- Email Extension Plugin
- Subversion Plugin (should be installed automatically)
Configure Plugins
Next to installing the plugins, they have to be configured. Therefore open the “Manage System” section as described above and
- configure your “MSBuild” plugin by choosing the msbuild.exe installed on your machine,
- configure the “Extended E-Mail Notification” plugin by setting the recipients and altering the subject and content (adding the environment variable $BUILD_LOG to the content will add the whole console output of the build and the test to the sent mail)
- and configure the “E-mail Notification” plugin by setting the SMTP server specific configuration.
Add New Job
Now, as the system is configured, we can add a new Jenkins job which will update the checked out files from a SVN repository, build both, the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send a mail with the report file attached.
First of all, create a new Job, choose “Build free-style software project” as job type and enter a name:
Configure Source Code Management
Next, we have to check out the source of both the application under test and our test automation project. Therefore choose Subversion as source code management tool, enter the repository holding your application under test as well as your test automation project and choose “Use ‘svn update’ as much as possible” as check out strategy:
With this configuration the application under test as well as the test automation project will be checked out and updated locally on the machine running Jenkins.
Add Build Steps
Now, as the source code management is configured, we can start with processing the updated files.
First of all, let’s add MSBuild steps for both projects:
Choose your configured MSBuild version and enter the path of the solution file relative to the repository root (which is the workspace folder of the Jenkins job) for both the automated and the automating project:
With adding these two build steps, the executables will be built automatically on the machine running Jenkins.
Now the newly built program should be tested.
This can be accomplished by adding a new “Windows batch command” starting the test suite executable (relative to the repository root which is the workspace folder root of the Jenkins job):
As you can see, some command line arguments are passed to the test suite executable.
In this sample the command line arguments “/zr”, which triggers the test suite executable to generate a zipped report file and “/zrf:.\Reports\Report-Build-%BUILD_NUMBER%.rxzlog” which defines the name and the location of the generated zipped report file, are used.
You can find a list of all available command line arguments in the section “Running Tests without Ranorex Studio” in our user guide.
The test suite executable returns “0” on success and “-1” on a failure. Based on this return value, Jenkins will mark the build as succeeded or failed.
Add Post-Build Action
After building and executing the application under test and the regarding test script, we will send an email which informs us about the success of the triggered build.
This email should include the zipped report file, mentioned before, as attachment.
Therefore add a new post-build action of type “Editable Email Notification”, choose the Report file location defined before as attachment, and add triggers for every job status you want to be informed about. In this sample an email will be sent if a job has failed or succeeded.
Run Job
After preforming these steps and saving your changes you can try out if everything works as expected by choosing “Build now”:
After running the generated job you will see all finished builds within the build hierarchy with icons indicating the status of the individual builds.
You can view the zipped report files of all builds by opening them via the local workspace (“Workspace/Reports”):
As configured before, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as attachment.
Add Repository Hook
After performing the steps mentioned above, we are able to manually trigger a build. But as we are working with subversion, it would be beneficial to trigger the script for each commit.
Therefore you can add a server side repository hook, which automatically triggers Jenkins to start a new build for each change committed as described in the subversion plugin documentation.
Alternatively you can activate polling of the source code management system as build trigger in your Jenkins job configuration.
As shown in following picture, you can define the interval, the source code management will be invoked (e.g. 5 minutes after every full hour):
Conclusion
Following the steps above you will be able to easily setup a continuous integration process performing the automated test of the application you develop.
After every commit a test automation is triggered that leads to automated feedback (success/failure) for the developer via mail with the Ranorex Report attached.
Please feel free to share your thoughts on this topic in the comments section.
Tags: Agile Software Developement, Command Line Arguments, Continuous Integration, Deployment, Hudson, Integration, Jenkins, Report, Source Control, Subversion, Test Suite
Subscribe
September 19th, 2012 at 1:42 pm
Can we have the same thing for TeamCity too, please?
September 19th, 2012 at 4:47 pm
Same as Ben, a similar thing for TeamCity would be awesome for myself as well!
September 19th, 2012 at 5:15 pm
Why not just run the windows service as an administrative user?
Additionally if you are using Tomcat on windows to run jenkins you just need to set the windows server to run as an administrative user.
September 19th, 2012 at 11:35 pm
Does Ranorex produce some kind of x-unit compatible xml reports? If so, we can use Jenkins graphing/reporting features to get great metrics.
September 20th, 2012 at 10:03 am
@Ben and Andrew
We will have a look at TeamCity…
@Steve
Thanks for your remarks.
@Ali
Basically you can try execute your Ranorex Tests using the NUnit framework as described in following blog posts:
http://www.ranorex.com/blog/combining-ranorex-3-0-and-visual-studio-2010-for-ui-unit-testing
http://www.ranorex.com/blog/using-nunit-for-test-execution
September 20th, 2012 at 10:49 pm
Hi Twalter,
Lets say I don’t want to refactor all my existing tests in Ranorex. Is there some command line way of exporting all test results to some XML file. I can then write a wrapper to pump those results into my reporting database
Thanks
September 24th, 2012 at 10:29 am
@Ali
Basically the generated *.rxlog file is a XML file (and the rxzlog file includes the *.rxlog file).
October 12th, 2012 at 2:49 pm
Has any one tried integrating ranorex with Hudson????
October 15th, 2012 at 9:54 am
@Ran
Correct me if I’m wrong, but as I can see using Hudson works the same way as using Jenkins.
October 22nd, 2012 at 8:44 am
[...] Integrating Ranorex Automation in Jenkins CI Process [...]
December 5th, 2012 at 9:52 am
I tried to run my ranorex testcase with Jenkins, and it failed with following error message, however, it works when I ran “C:\Program Files (x86)\Jenkins\jobs\Test\workspace>.\test\test\bin\debug\test.exe /tc:TestCase /zr /zrf:.\Report\build-3.rxzlog”
error message:
[2012/12/05 16:17:42.911][Debug ][Logger]: Console logger starting.
[2012/12/05 16:17:43.052][Info ][Test]: Test Case ‘TestCase’ started.
[2012/12/05 16:17:43.052][Info ][Test]: Test Module ‘Recording1′ started.
[2012/12/05 16:17:43.067][Info ][Keyboard]: Key sequence ‘{LWin down}r{LWin up}ca’.
[2012/12/05 16:17:45.486][Info ][Keyboard]: Key sequence ‘lc{Return}’.
[2012/12/05 16:17:52.476][Info ][Mouse]: Mouse Left Click item ‘Calculator.Calc.Button134′ at 9;8.
[2012/12/05 16:17:52.538][Warn ][Java AWT/Swing]: Please make sure that you have correctly instrumented your Java installation or application. Otherwise Ranorex is not able to access UI objects inside your Java AWT/Swing application.
The following website provides more information on this technology limitation:
http://www.ranorex.com/support/user-guide-20/technology-instrumentation/testing-of-java-applications.html
(This message is only shown once per report.)
[2012/12/05 16:18:22.546][Error ][Module]: No element found for path ‘/form[@title='Calculator']‘ within 30s.The folder ‘testRepository.Calculator’ was not found.Failed to find item ‘testRepository.Calculator.Calc.Button134′.
[2012/12/05 16:18:22.718][Failure][Test]: Test Module ‘Recording1′ completed with status ‘Failed’.
[2012/12/05 16:18:22.718][Failure][Test]: Test Case ‘TestCase’ completed with status ‘Failed’.
[2012/12/05 16:18:22.843][Debug ][Logger]: Console logger stopping.
C:\Program Files (x86)\Jenkins\jobs\Test\workspace>exit -1
key couldn’t be input even if testcase starts in log, so calculator didn’t show up.
December 6th, 2012 at 9:23 am
@Alin
Make sure that you have started Jenkins from command line and not as a service as described in the infrastructure section of the bog post.
December 7th, 2012 at 10:13 am
Hi twalter,
Thanks for your reply.
it works well if start Jenkins from command line.
and I have other question:
the test suite name can be anything even if nothing by “test.exe /ts:”, it also can be ran.
is that expected behavior?
thanks again!
December 13th, 2012 at 12:24 pm
@Alin,
the command line argument “testsuite|ts” can be used if you have more than one test suite files (*.rxtst) describing your test suite project. As the help text says, if nothing else is defined, the test suite file having the same name as the executable or the first test suite file in the same folder will be taken.
May I ask what you are going to perform in detail?
April 26th, 2013 at 9:47 am
Hello,
I have successfully configured Jenkins to execute my Ranorex sutie and a Virtual Machine.
It runs fine however when it comes to using the data connector it fails in locating the required file.
[2013/04/25 17:20:35.894][Info ][Test]: Test Case ‘LocationsAndDepartments’ started.
[2013/04/25 17:20:35.896][Info ][Test]: Test Module ‘Locations’ started.
[2013/04/25 17:20:42.368][Info ][User]: Checking the Visility of the buttons…
[2013/04/25 17:20:42.370][Success][Validation]: ‘{Button:Open Tree}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.372][Success][Validation]: ‘{Button:Close Tree}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.375][Success][Validation]: ‘{Button:Configure Company Details}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.377][Success][Validation]: ‘{Button:Modify}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.378][Success][Validation]: ‘{Button:Delete}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.380][Success][Validation]: ‘{Button:AddLocation}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.382][Success][Validation]: ‘{Button:AddDepartment}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.383][Success][Validation]: ‘{Button:AddUser}’ ‘Visible’ is being checked
[2013/04/25 17:20:42.383][Info ][User]: Checking the Enabled state of the buttons…
[2013/04/25 17:20:42.384][Success][Validation]: ‘{Button:Open Tree}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.385][Success][Validation]: ‘{Button:Close Tree}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.385][Success][Validation]: ‘{Button:Configure Company Details}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.386][Success][Validation]: ‘{Button:Modify}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.387][Success][Validation]: ‘{Button:Delete}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.388][Success][Validation]: ‘{Button:AddLocation}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.389][Success][Validation]: ‘{Button:AddDepartment}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.390][Success][Validation]: ‘{Button:AddUser}’ ‘Enabled’ is being checked
[2013/04/25 17:20:42.390][Info ][User]: Checking the edit box objects…
[2013/04/25 17:20:45.674][Success][Validation]: ‘{Text:nameInput}’ ‘Enabled’ is being checked
[2013/04/25 17:20:45.675][Success][Validation]: ‘{Button:Cancel}’ ‘Enabled’ is being checked
[2013/04/25 17:20:45.677][Success][Validation]: ‘{Button:Create}’ ‘Enabled’ is being checked
[2013/04/25 17:20:47.385][Success][Test]: Test Module ‘Locations’ completed with status ‘Success’.
[2013/04/25 17:20:47.386][Info ][Test]: Test Module ‘LocationsDepsTreeBehaviour’ started.
[2013/04/25 17:20:47.396][Error ][Module]: The Excel file with name ‘Traffic Live Automation Data.xlsx’ could not be found.
[2013/04/25 17:20:47.422][Failure][Test]: Test Module ‘LocationsDepsTreeBehaviour’ completed with status ‘Failed’.
[2013/04/25 17:20:47.422][Failure][Test]: Test Case ‘LocationsAndDepartments’ completed with status ‘Failed’.
If I execute the Test Suite from the command line it locates it correctly. It is only when using this Batch Call from jenkins does it generate this error. Do I need to pass in a Global Parameter or is there any directove I need to pass to the batch command?
The command is as follows:
.\TrafficLIVE\bin\Release\Traffic_LIVE.exe
The required file is located in the .\TrafficLIVE\bin\Release directory so I am at a bit of a loss.
Any help would be appreciated.
Thanks,
Chris
April 26th, 2013 at 9:56 am
@Chris
try to change the working directory from your home directory to the “Release” directory before starting the test automation.
The regarding command should look as follows:
cd .\TrafficLIVE\bin\Release
Traffic_LIVE.exe
Regards,
Tobias
April 26th, 2013 at 10:11 am
@Tobias
That was exactly my problem. Thank you very much for your swift reply.
Regards,
Chris