Running Ranorex Automated Tests with Microsoft Test Manager

Using Microsoft Team Foundation Server, you can cover your whole ALM process from source control over automated builds to quality assurance. This blog post illustrates how to use the Ranorex test automation framework to implement coded UI tests that execute Ranorex based automation code or even whole Ranorex test suites using Microsoft Test Manager.

• Infrastructure
• Implementing Coded UI Tests to Execute Ranorex Automation
• Conclusion

First of all, you need a working infrastructure to run your Coded UI Test with integrated Ranorex automation. Therefore, please have a look at the Microsoft documentation regarding Team Foundation Server, Microsoft Test Manager, and Coded UI Tests or this getting started blog post.

A typical infrastructure for executing your ‘Coded UI Tests’ might look something like this:

TFS infrastructure

In the mentioned infrastructure the Coded UI Test will be implemented on the Visual Studio workstation. This Coded UI Test will kick off the Ranorex automation.

After implementing the test method, it can be checked into source control on the TFS.

The implemented Coded UI Test can than be added to a TFS test case as the associated automation process as described in this article.

To make the TFS test case accessible to MTM, it has to be built and deployed by defining a Build Definition that triggers a Build Controller to build the Coded UI test project and to deploy it to a network drive.

After doing so, the test case should be available in MTM and can be executed on the Test Agent machine using the Test Controller.

Cancel Coded UI Test generation

The new UI Test should contain a test method looking similar to this:

[TestMethod]
public void CodedUITestMethod1()
{
}

Starting from this point you can either implement your own Ranorex code within your test methods or start a whole Ranorex test suite generated with Ranorex Studio from your test methods.

No matter which approach you are going to follow, it’s necessary to add references to the Ranorex libraries and add a “using”-directive for the Ranorex namespace as described in the “Visual Studio Integration”-section of our user guide.

After adding the references and the using-directive, you can simply start writing your Ranorex automation code as usual.

The recommended way of adding Ranorex automation to a Coded UI Test is to start a whole Ranorex test suite or specific test cases – generated with Ranorex Studio – from your test method.

For demonstration purposes, you can simply create a new KeePassTestSuite sample project from the Ranorex Studio start page as shown in following figure:

Create a new KeePassTestSuite solution

As we are going to start our test suite directly from our test method, there is no need to start it from the Program.cs of the KeePassTestSuite project any more. Thus, you can delete the whole main method from Program.cs.

When running the test suite using the TestSuiteRunner class, we need to specify a container type which is in the project containing the test suite. We will use the Program class as the container type and, consequently, we have to make that class public. Your Program.cs file should look like this:

...
namespace KeePassTestSuite
{
    public class Program
    {

    }
}
...

Change the Output type of the generated project to “Class Library” and add the project to your Coded UI Test solution in Visual Studio.

After doing so, add a project reference to the just added KeePassTestSuite project to the Coded UI Test project.

Added the Ranorex references and the sample Ranorex test suite reference to the Coded UI Test project

To execute the application under test, it is necessary to copy all needed files to the test machine on which the test will be executed.

This is done using deployment items which define the file and the relative directory where the file should be deployed to. So add a deployment item for each file which has to be copied.

Additionally, you have to include the KeePass folder in your project and to make sure, that all files needed for executing the KeePass application are copied to the output folder during the build process.

This can be done by setting the option “Copy to Output Directory” to “Copy always” in the properties of each KeePass-specific file in the KeePassTestSuite project added as deployment item in code.

Copy the added deployment items to the output directory

...
[TestMethod]
[DeploymentItem("KeePassTestSuite.dll")]
[DeploymentItem("KeePassTestSuite.rxtst")]
[DeploymentItem("KeePassTestSuite.rxtmg")]
[DeploymentItem("Passwords.csv")]
[DeploymentItem("KeePass/KeePass.exe", "./KeePass")]
[DeploymentItem("KeePass/NewDatabase.kdbx", "./KeePass")]
[DeploymentItem("KeePass/KeePass.exe.config", "./KeePass")]
[DeploymentItem("KeePass/KeePass.config.xml", "./KeePass")]
[DeploymentItem("KeePass/KeePass.XMLSerializer.dll", "./KeePass")]
[DeploymentItem("KeePass/KeePassLibC32.dll", "./KeePass")]
[DeploymentItem("KeePass/KeePassLibC64.dll", "./KeePass")]
[DeploymentItem("KeePass/XSL/KDB4_DetailsFull.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_DetailsLite.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_PasswordsOnly.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_Styles.css", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_Tabular.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/TableHeader.gif", "./KeePass/XSL")]
public void CodedUITestMethod1()
...

Note: If you are going to have more than one test methods having the same deployment items in your Coded UI test class, you can add the deployment items to the class instead of the test method. Therefore just add the deployment items directly after the line [CodedUITest] instead of [TestMethod].

Now we can run the test suite from the “Coded UI Test” test method using the following code snippet:

...
public void CodedUITestMethod1()
{
    Ranorex.Keyboard.AbortKey = System.Windows.Forms.Keys.Pause;

    int error = TestSuiteRunner.Run(typeof(KeePassTestSuite.Program), @"/pa:GlobalExecutionPath=.\KeePass\KeePass.exe /zr /zrf:report.rxzlog");
    TestContext.AddResultFile("report.rxzlog");

    if (error != 0)
        throw new RanorexException("Test run failed! See report.rxzlog for further details!");
}
...

The first line of the test method will set the abort key which allows you to stop the automation script on the system under test.

Command Line Arguments

The next line will start the test suite with the given command line arguments.

In our case the global test suite parameter “GlobalExecutionPath” is set to the relative path the KeePass application has been deployed to.

The argument “/zr” will trigger the test suite to generate a zipped report which makes it easier to attach the report to the test case executing the test method.
The argument “/zrf” allows you to specify the name of the zipped report.

With command line arguments you can e.g. only start a single test case (“/tc” argument) or start a defined run configuration (“/rc” argument).
You can find a list of all available command line arguments in the  section “Running Tests without Ranorex Studio“ in our user guide.

The next line of code will attach the zipped report file to the test results, which allows you to analyze the test run directly from MTM.

The if statement will throw an exception if the test suite execution fails, which will make the test case fail, too.

After checking in the test and building the solution, the test method can be executed on the system under test by triggering the corresponding test case via MTM.

Execute Coded UI Test with Ranorex test suite

As mentioned before, if the test suite run produces errors, an exception will be thrown which as well marks the test run triggered by MTM as failed.

In the following figure you can see that the zipped report is attached to the test result, which allows you to analyze the error.

Failed Coded UI Test with attached Ranorex report

The zipped report can be opened directly from MTM.

Attached Ranorex report

The zipped report will be attached to the test result for successfully passed test runs as well (see following figure).

Passed Coded UI Test with attached Ranorex report

After learning how to start a whole Ranorex test suite from a Coded UI test method, let’s see how to start specific test cases from test methods.

Like before, if we are going to have more than one test method within our test class, we can add the deployment items directly to the class instead of adding them to each and every test method.

By adding a new command line argument to our code, we can specify which test case should be triggered.

Following sample code will show how to generate a test method for each test case defined by the KeePass test suite.

...
[CodedUITest]
[DeploymentItem("KeePassTestSuite.dll")]
[DeploymentItem("KeePassTestSuite.rxtst")]
[DeploymentItem("KeePassTestSuite.rxtmg")]
[DeploymentItem("Passwords.csv")]
[DeploymentItem("KeePass/KeePass.exe", "./KeePass")]
[DeploymentItem("KeePass/NewDatabase.kdbx", "./KeePass")]
[DeploymentItem("KeePass/KeePass.exe.config", "./KeePass")]
[DeploymentItem("KeePass/KeePass.config.xml", "./KeePass")]
[DeploymentItem("KeePass/KeePass.XMLSerializer.dll", "./KeePass")]
[DeploymentItem("KeePass/KeePassLibC32.dll", "./KeePass")]
[DeploymentItem("KeePass/KeePassLibC64.dll", "./KeePass")]
[DeploymentItem("KeePass/XSL/KDB4_DetailsFull.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_DetailsLite.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_PasswordsOnly.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_Styles.css", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/KDB4_Tabular.xsl", "./KeePass/XSL")]
[DeploymentItem("KeePass/XSL/TableHeader.gif", "./KeePass/XSL")]
public class CodedUITest1
{
    public CodedUITest1()
    {
    }
    public void StartRanorexTestCase(string tc)
    {
        Ranorex.Keyboard.AbortKey = System.Windows.Forms.Keys.Pause;

        int error = TestSuiteRunner.Run(typeof(KeePassTestSuite.Program), @"/pa:GlobalExecutionPath=.\KeePass\KeePass.exe /zr /zrf:report.rxzlog /tc:" + tc);
        TestContext.AddResultFile("report.rxzlog");

        if (error != 0)
            throw new RanorexException("Test run failed! See report.rxzlog for further details!");
    }

    [TestMethod]
    public void AddEntry()
    {
        StartRanorexTestCase("TC_AddEntry");
    }

    [TestMethod]
    public void AddEntryFromCode()
    {
        StartRanorexTestCase("TC_AddEntryFromCode");
    }

    [TestMethod]
    public void AddEntryWithParams()
    {
        StartRanorexTestCase("TC_AddEntryWithParams");
    }

    [TestMethod]
    public void ValidateVersioNumber()
    {
        StartRanorexTestCase("TC_ValidateVersionNumber");
    }

    [TestMethod]
    public void OpenLoginAndClose()
    {
        StartRanorexTestCase("TC_OpenLoginAndClose");
    }
...

After adding new test cases for each test method, you can run these test cases separately from MTM.

Coded UI Test Methods for each Ranorex test case

Tags: Agile Software Developement, Code Samples, Coded UI Test, Command Line Arguments, Continuous Integration, Deployment, Integration, Microsoft Test Manager, Report, Source Control, Team Foundation Server, Test Suite, Visual Studio