Running tests | Ranorex
Help CenterUser GuideRanorex Studio fundamentals

Run tests

 

In this chapter you’ll learn about the various options to control test runs and how to run tests out of the test suite.

In this chapter

    Sample solution

    The explanations in this chapter are based on a sample solution. You can download it below.

    Sample solution

    Theme: Test suite run
    Time: Less than 10min

    Download sample file

    Installation:

    Unzip to any folder on your computer.
    Start Ranorex Studio and open the solution file RxDatabase.rxsln

    tipp icon

    Hint

    The sample solution is available for Ranorex versions 8.0 or higher. You need to agree to the automatic solution upgrade for versions 8.2 and higher.

    Run configurations

    You can include and exclude test cases and smart folders from a test run. To do so, simply check or uncheck them in the test suite.

    Selection checkboxes for test suite elements

    Checkboxes to include/exclude from test run

    The current state of checked/unchecked test cases and smart folders is called a run configuration. You can save run configurations for reuse and switch between them using the drop-down menu in the test suite view.
    Manage test run configurations

    Manage run configurations

    Click the run configurations drop-down menu.

    Click Manage run configurations…

    Add a new run configuration

    Adding a new run configuration

    Click Add.

    Give the new run configuration a meaningful name.

    Click OK.

    Result(s):

    • The run configuration can be selected from the drop-down menu.
      Example for different run configurations

       Different run configurations

      TestRun configuration including all test suite elements
      TestWithoutValidation configuration excluding the validation smart folder

      Disable/enable test suite items

       

      Similarly to including/excluding test cases and smart folders, you can also enable and disable recording modules, code modules, module groups, and setup and teardown regions. Disabled items are not executed during a test run.

      Disabling a test suite item

      Disabling a recording module

      Right-click the item you want to disable.

      Click Disable. The item will be grayed out.

      Note icon

      Note

      • Disabling a module group disables all modules in this module group.
      • Disabling a setup/teardown region disables all items in it.

      Run iterations

      By default, test cases and smart folders are run once during test execution. However, you may want to run them several times. You can do so with run iterations.

      Specifying run iterations

      Configuring run iterations

      Right-click the test case or smart folder you want to iterate.

      Click Properties. The Properties pad appears to the right of the test suite.

      Next to Iteration count, set the desired number of iterations.

      Iteration count in the Properties.

      Result(s):

      • The number of iterations appears next to the test suite item.
      • In our example, the test case is iterated 5 times.
      Iteration count information in test suite view

      Number of iterations in test suite view

      Auto-retry

      In UI testing, errors will sometimes occur simply because the application under test was unresponsive. In these cases, one solution is to simply re-run parts of the test. You can do so with auto-retry. Test cases or smart folders with an auto-retry count will be rerun until they are either successful or all retries have been used up.

      Auto-retry function for test suite elements

      Configuring auto-retry

      Right-click the test case or smart folder you want to set up auto-retry for.

      Click Properties. The Properties pad appears to the right of the test suite.

      Next to Retry count, set the desired number of retries.

      Retry count in the Properties

      Note icon

      Note

      If there are data bindings or run iterations, the retries will start at the point of failure. For example, if the failure occurred at iteration 3 of 5, that’s where the retry will start

      Only test cases or smart folders that failed every single retry will be marked as failed in the report.

      Report levels

      You can also set the report level of test cases and smart folders in the test suite. Report levels allow you to control what is displayed in the report and where. This is especially useful for complex tests with many test cases and smart folders, to keep the report structured.

      Further reading

      Report levels are beyond the scope of this chapter. They are explained in Ranorex Studio fundamentals > Reporting > ⇢ Concept of report levels.

      Error behavior

      Errors are part of testing.  This is why it’s important to tell Ranorex what to do when an error occurs. You can do so by setting up the error behavior of test cases and smart folders in the test suite.

      Setting up error behavior

      The error behavior is set up in the context menu of a test case or a smart folder. There are four different types, explained below.

      Specification of error behavior

      Specification of error behavior

      tipp icon

      Hint

      The default error behavior is Continue with sibling.

      Continue with iteration

      Error behavior: Continue with iteration

      Error behavior: Continue with iteration

      Continue with iteration

      The test run will continue with the next iteration of the smart folder Validation.

      Continue with sibling

      Error behavior: Continue with sibling

      Error behavior: Continue with sibling

      Continue with sibling
      The test run will continue with the next sibling test case or smart folder. In our case, this is the smart folder DatabaseCleanUp.

      Continue with parent

      Error behavior: Continue with parent

      Error behavior: Continue with parent

      Continue with parent

      The test run will continue with the next parent test case or smart folder. In our case, this is the test case CompleteDatabaseTest.

      Stop

      The test run is stopped entirely immediately.

      Error behavior of setup/teardown regions

      Setup and teardown regions follow a special, fixed error behavior.

      • An error in the setup region immediately stops the test.
      • If a module fails in the teardown region, the next module is run.

      Run a test from the test suite view

      Running a test

      Running a test

      Click RUN.

      Watch Ranorex start up the test.

      Pay attention to the progress information during the test.

      tipp icon

      Attention

      After you’ve clicked RUN, do not use the keyboard or mouse. Doing so would interfere with the test actions and cause a test failure.

      Result(s):

      • Once the test run has finished, the report will appear.
      Final test report summary

      Further reading

      Reports are introduced and explained in Ranorex Studio fundamentals > Reporting > ⇢ Introduction.