Create and execute an ADO pipeline | Ranorex Studio User Guide
Help CenterUser GuideInterfaces and connectivity

Create and execute a pipeline

Pipelines in Azure DevOps are how you build, test, and deploy your code. On this page, you’ll find out how to create, configure, and execute pipelines in Azure DevOps to run your Ranorex Studio solution. For more details about pipelines, please refer to the official Azure Pipelines documentation.

In this chapter

    Create a pipeline

    To create a new pipeline for your Azure DevOps project:

    In your opened project, click Pipelines in the register on the left.

    Click New pipeline.

    Depending on whether you want to specify a pipeline using YAML or on your own:

    Click Azure Repos Git (YAML) (recommended for experts, not covered here).

    Click Use the classic editor.

    Configure as shown below and click Continue.

    In the next step, click Apply for .NET Desktop. This is the framework Ranorex Studio solutions are based on.

    Pipeline overview

    The pipeline overview shows the various components a pipeline consists of and offers many possibilities for configuration. Here we only explain what’s relevant for the Ranorex Studio integration.

    Configuration categories of a pipeline. In the image, Tasks is selected.

    Basic pipeline configuration. Here you can give the pipeline a name, set the agent pool, and set optional parameters.

    Here you can configure the sources (i.e. repositories) the pipeline uses. This corresponds to what we did in Create a pipeline above.

    A pipeline always contains at least one agent job. This is the logical set of tasks that the pipeline carries out.

    An agent job always contains at least one task. Tasks are carried out in the order they appear in under the agent job.

    The configuration window. Configuring one of the above items brings up this window. In the image, the window for configuring the agent job is shown.

    Basic pipeline configuration

    Under Tasks, click Pipeline to bring up the basic pipeline configuration, where you can make the following settings:

    Here you can change the pipeline name.

    Here you select the agent pool for the pipeline. You can use the default local pool or choose from a range of hosted pools that have various images already set up.

    Here you can set optional pipeline parameters. The preset parameter in the image builds the executable from the Visual Studio solution file (.sln file ending).

    Pipeline tasks for Ranorex Studio tests

    For Ranorex Studio tests to be executed correctly in an Azure DevOps pipeline, the agent job must contain the tasks listed below.

    Note icon

    Note

    The .NET pipeline already contains various default tasks. Do not delete them.

    tipp icon

    Hint

    Instead of the following tasks, you can also use the official Ranorex Studio build task, explained below after this topic.

    Build the Ranorex Studio solution

    The task Visual Studio build builds the Ranorex Studio test solution. It is already included by default in the agent job and functional.

    Configuration

    Change the display name for easier identification.

    Change the platform to x86, if required.

    Execute the build

    To execute the build of the Ranorex Studio solution, you need to add the Command line task to the agent job.

    To do so:

    Click the + symbol next to the agent job.

    In the search bar, enter command line.

    Click Add.

    Note icon

    Note

    Make sure the Command line task comes after the Visual Studio build task. Otherwise, execution will fail.

    Configuration

    Change the Display name for easier identification.

    Replace the default script with the following:

    System variable for the current working directory of the Azure DevOps project.

    Folder for the Ranorex Studio solution.

    Folder for the Ranorex Studio project in the solution.

    Default output folder for Visual Studio projects.

    File name of the executable build. You can add further optional arguments to the .exe.

    Reference

    For a full list of command line arguments and how to use them, refer to

    Ranorex Studio expert > Runtime and remote execution > ⇢ Command line execution

    Reporting

    When running a Ranorex Studio test through an Azure DevOps pipeline, the associated Ranorex Studio report is not saved to the working directory of the build by default. To include it, you have to add a Publish build artifacts task in the agent job. Then:

    Click the Publish build artifacts task.

    Change the display name, if desired.

    Change the directory as shown in the image.

    Change the artifact name, if desired.

    Result

    Once a pipeline has finished, you can see the report in the summary and open it from there.

    Click Pipelines > Builds.

    Click Summary.

    Under Build artifacts published, click the item. It will have the name you specified in the task.

    Show test results in the build results

    By default, the results of a Ranorex Studio test are not displayed in Azure DevOps. You can however have them displayed by using the JUnit format of Ranorex Studio reports.

    To do so:

    To the agent job, add a Publish test results task.

    Change the task’s display name, if desired.

    Set the test result format to JUnit.

    As test results files, specify **/*.xml.

    As search folder, specify the current working directory.

    Switch to the Command line task and extend the script as in the image:

    The executable build of the Ranorex Studio solution.

    Argument to create the report in the JUnit format.

    Argument to zip the report.

    Argument to specify a directory and file name for the zipped report.

    Reference

    For a full list of command line arguments and how to use them, refer to

    Ranorex Studio expert > Runtime and remote execution > ⇢ Command line execution

    Result

    Under Pipelines > Builds > Tests, you can see various test details after a build has been executed.

    Official Ranorex Studio build task

    The official Ranorex Studio build task collects all the tasks described above under Pipeline tasks for Ranorex Studio tests and more in one simple task with a straightforward UI to configure it. You can download it as a free extension from the Azure DevOps Marketplace here.

    Simply add the task to a job to get started.

    Note icon

    Note

    The build task creates a set of command line arguments based on how you configure it. Therefore, it behaves in the same way as running a Ranorex Studio test from a command line interface and the arguments themselves also work the same way.

    All command line arguments are explained  ⇢ here.

    Display name of the task.

    Task settings controlling the integration with Ranorex Studio. We explain these below.

    General settings

    The relative path from the repository folder to the test executable in the output folder of your Ranorex Studio solution. Required.

    The relative path from the repository folder to a specific test suite file (.rxtst) in the output folder of your Ranorex Studio solution. Useful if your test contains several test suites and you only want to run one of them.

    The run configuration that will be used for the test run. Enter exactly as it is named in the run configuration manager in Ranorex Studio. Useful if you have more than one run configuration.

    Reporting

    Enables reporting, i.e. a Ranorex Studio report will be attached to the build.

    Allows you to enter a custom report name. You can use report-specific placeholders, explained

    ⇢ here under Reporting.

    The minimum report level that events must have to be included in the report. (Default) uses the report level setting that was active when the solution was built in Ranorex Studio. Report levels are explained ⇢ here.

    TestRail integration

    These options allow you to pass test results to a TestRail instance.

    Enables reporting test results to TestRail.

    Enter the service connection for your TestRail instance. If you don’t have one yet, you need to configure it in Azure DevOps under Project Settings > Service connections > New service connection > Generic. There, enter your TestRail credentials. You can also click the Manage link next to the field to get to the Service connections menu.

    The identifier of an existing test run in TestRail to report test results to. If you don’t specify an ID or it doesn’t exist, results will be reported to a new test run.

    Creates a new test run in TestRail with the specified name to report test results to. Without this argument the default name is used.

    Advanced

    Here you can specify additional command line arguments to customize the test run. All available command line arguments are explained ⇢ here.

    Execute a pipeline

    To execute a pipeline, you need a  ⇢ configured Azure Pipelines agent.

    Click Pipelines > Builds.

    Click Run.

    In the dialog that appears, review the pipeline settings and click Run.

    Pipeline progress view

    The pipeline will carry out all tasks in the specified order. You can view the progress in Azure DevOps.

    Indicates that a pipeline has been started.

    Start time and elapsed time.

    Completed tasks.

    Current task, including debug info.

    Agent window

    Once you’ve started a pipeline, the agent job is passed to the agent, which then runs it. The agent window will show that a job has been started.

    Ranorex Studio test run

    The agent runs the Ranorex Studio test as if it were started from within Ranorex Studio.

    Pipeline completion

    When a pipeline has completed, it will be indicated in the pipeline overview.

    List of all created pipelines for the current project.

    List of all builds in the selected pipeline, including execution stats (success/failure).

     

    Click the build to display the results view for it.

    After completing a job, the agent shows the result and then returns to idling.

    Trigger a pipeline automatically

    You can also trigger pipelines automatically. To do so, you need to enable the option Enable continuous integration. Any changes to the Ranorex Studio solution committed and pushed to Git will then trigger the respective pipeline.

    To enable this option:

    Click Pipelines, select the pipeline, and click Edit.

    Click Triggers.

    Check Enable continuous integration.