Posted by Tobias Walter on Wednesday, November 14th, 2012 at 4:00 pm to Integration

The common method for developing software has been shifting more and more from classical development models to more agile software development. In response to numerous inquiries from Ranorex customers, we have decided to write another Ranorex integration blog. This blog post will illustrate the integration of Ranorex Automation into a Bamboo Continuous Integration Process.

Bamboo Integration

Infrastructure

The fundamental principle of agile software development is to control changes and rapidly get response about the effects of the changes.
Applied to your Continuous Integration Process the infrastructure can be sketched as follows:

Bamboo Integration

A developer or test engineer commits his changes to the version control system. Based on the committed change, the version control system triggers the continuous integration system to start a job which triggers the build agent to

  • check out the modified source from the version control system,
  • build the application under test (AUT),
  • build the Ranorex Test Suite and
  • execute the Ranorex Test Suite which tests the AUT.

The corresponding Ranorex Report file as well as a build log will be attached to the build iteration.
The developer and/or test engineer responsible for the changes will be notified about the build and test success.

Version Control System

Bamboo provides built-in support for source control systems like Subversion, Mercurial, Git, Perforce and CVS. For this blog we are going to use Subversion as version control of choice.

In this sample we have two solutions placed in our repository: the application we are going to test and the testing application.

Repository-Structure

To start the application under test from your test project, simply add a new ‘Run Application’ action to your recorder action table which starts the application under test using a relative path from the bin folder of the testing application.

Add RunApplication Action

Bamboo Server

Download the Bamboo installation package and follow the installation instructions.
A download link and further information about Bamboo can be found on the
Atlassian homepage.

If you are going to use the default build agent running locally on the server, make sure to not start the Bamboo Server as service, but to start the server in a console as you can see in following figure. The reason is that a service does not have sufficient rights to start and access UI applications.

Start Bamboo Server From Console

After starting the server in a console, you can access the web interface of your server from the address shown in the console window (e.g. localhost:8085).

Follow the instructions of the Setup Wizard to configure your CI server.

To allow your CI process to notify the responsible developers and testers about a build success, configure your mail server (Administration -> Communication -> Mail Server).

Bamboo Configure Mail Server

Build Agent

The default installation of the Bamboo Server includes a build agent running on the local machine. You can scale out your build infrastructure on multiple machines by installing several remote build agents using following instructions: Bamboo Remote Agent Installation Guide

Note: Make sure that a valid Ranorex license as well as the Ranorex main components are installed on every Build Agent you are going to build/execute Ranorex automation code.

Add a new Plan

After setting up all necessary components, it’s time to add a new Bamboo plan.

For this purpose click the ‘+ Create Plan’ button in the upper right and choose ‘Create a New Plan’.

Bamboo Create Plan 00

Choose a project the plan will be added to and a plan name.

Bamboo Create Plan 01

Version Control Settings

Set up the path to your source repository and the build strategy.

Bamboo Create Plan 02

Configure Tasks

After automatically adding the first task which checks out the source code, add a MSBuild task for building the application under test by choosing the solution file (*.sln) of the application under test relatively to the SVN working copy root.

Bamboo Configure Tasks 01

Bamboo Add MSBuild Task-01

Bamboo Add MSBuild Task 02

Add a MSBuild task for building the testing application by choosing the solution file (*.sln) of the Ranorex Automation project relatively to the SVN working copy root.

Bamboo Configure Tasks 02

Bamboo Add MSBuild Task-01

Bamboo Add-MSBuild Task 03

Add a command task which starts the Ranorex Test Suite executable.

Bamboo Configure Tasks 03

Bamboo Add Command Task 01

Bamboo Add Command Task 02

As you can see, we defined a set of command line arguments.

In this sample the command parameter ‘/zr’, which triggers the test suite executable to generate a zipped report file and ‘/zrf:.\Reports\Report.rxzlog’ which defines the name and the location of the generated zipped report file, as well as the command parameter ‘tc:/TestCase’ which specifies the test case to be executed are used. The location of the report file will later be used to define the build artifacts.

You can find a list of all available command line arguments in the section ‘Running Tests without Ranorex Studio‘ in our user guide.

The test suite executable returns ’0′ on success and ‘-1′ on a failure which allows the test agent to either mark this build step as successfully or failed.

After defining the tasks, the newly added plan can be enabled and created.

Bamboo Configure Tasks 04

This will trigger an initial build.

Bamboo Initial Build

Configure Artifacts

To attach a Ranorex Report to every build (= test run), you have to create an artifact definition, specifying the path and the name of the files to attach to the build.
For this purpose click ‘Configure Plan’ in the plan’s ‘Actions’ menu.

Bamboo Configure Plan

In the plan’s configuration choose the default job and the ‘Artifacts’ tab and click on the ‘Create Definition’ button.

Choose a name for your definition and define the path where the Report files will be stored, relatively to the SVN working copy root (e.g. ./TestCIProject/TestCIProject/bin/Debug/Reports/). If you want to attach all files in that path to your build, you can use the copy pattern ‘*.*’. Otherwise you have to use a more specific copy pattern like ‘*.rxzlog’ which would only copy zipped report files.

Bamboo Edit Artifact Definition

Configure Notifications

To notify the responsible test engineer and/or developer about the build status, you can add email notifications. For this purpose open the ‘Notifications’ tab in the ‘Plan Configuration’.

There you can add several kinds of build notifications.

Bamboo Add Notification

Run Job

As the configuration of the plan is now finished, you can try out if everything works as expected by committing a change to either your application under test or your Ranorex project. The build will automatically be triggered and a notification will be sent to the user or email address you specified.

Bamboo Successfully Build

You can investigate the attached Report file of a specific build by opening the ‘Artifacts’ tab of the build and clicking the Report file.

Bamboo Artifact

Conclusion

With this step by step instruction you will be able to easily setup a Bamboo CI process executing an automated test for each committed change, thereby providing rapid feedback to the responsible test engineer and/or developer about the build success.

Please feel free to share your thoughts on this topic in the comments section.

Tobias Walter

Online Marketing and Content Specialist at Ranorex
Tobias Walter is Online Marketing and Content Specialist at Ranorex GmbH. Next to his marketing activities he also writes articles published in severals blogs and journals.

Latest posts by Tobias Walter (see all)

Share

Tags: , , , , , , , , ,

5 comments

  1. Integrate Ranorex into Any Continuous Integration Process | Ranorex Blog

    [...] Bamboo CI with Ranorex Test Automation [...]

  2. Mikko Korhonen

    Hi!

    I’ve been trying out Bamboo with Ranorex, and followed you’re blogpost on running tests. The problem here is, when the Ranorex tests fail, we get a “No failed tests found, a possible compilation error occurred.” error message for the build. Do you know a possible way to get Bamboo recognise Ranorex test runner in such a way it could inform that it’s just the Ranorex tests that failed, not the actual compilation of the project? As it is now it makes it too much of a hassle to find out where the failure originated from, even though it does state somewhere in thousands of lines of logs, that one Ranorex test case failed. Writing our own plugin just for this isn’t an option. Aside from this one issue, a great blog post!

  3. twalter

    @Mikko
    Basically a failing Ranorex Test forces the Ranorex Test Suite executable to return the value “-1″. This forces the added Command Task to Fail, as I can say from my own experience with Bamboo. That worked pretty good for me. Maybe the following link might help you solving your issue:
    All the tests passed, but bamboo build fails with a statement “No failed tests found, a possible compilation error occurred.”.
    Regards,
    Tobias

  4. Mikko Korhonen

    @twalter
    Yes, I understood that much, but the thing was that for someone else looking at the failing build besides me, it might not be obvious it’s the tests that failed. If only Bamboo could somehow separate the Ranorex command from the other tasks, cause if the Ranorex tests fail I want it to be signalled as failing the build, just more clearly what failed it. Thank you for the link, though it only helped a little. I was able to run Ranorex tests earlier trough a script/bat file so that the result didn’t show up in Bamboo at all. Would it then be possible to parse the Ranorex log with Bamboo tasks to find out the result and consider it as a test? As in the link there was some talk of Junit parser going trough an .xml file.

  5. twalter

    @Mikko
    if you do not want to make a build fail when a test fails simply add a “Script” task instead of a “Command” task which starts your executable same way as the command does. Then simply add the line “exit 0″ to the script. This will set the return value to success independently the return value of the Test Suite executable. So even the test fails the script will return a successful execution. Following this approach, you have to look into the Report to see if the Test ran successfully.
    Regards,
    Tobias

Leave a Reply