Running tests | Ranorex
HomeUser GuideRanorex Studio FundamentalsRunning tests

Running tests

This chapter introduces and explains the various settings and configurations to control test runs and explains how to run tests out of the test suite.

In this chapter

    Sample solution

    The explanations herein are based on a sample solution which can be downloaded hereinafter.

    Training! What do I do?

    Theme: Test suite run
    Time: Less than 10min

    Download sample file

    Installation:

    1. Unzip the project directory to any folder on your computer
    2. Start Ranorex Studio and open the solution file RxDatabase.rxsln

    Test run specification

    Test cases and smart folders can be selected to be included in the test run, or deselected to be excluded from the test run. Check test suite elements to include to test run/deselect to exclude them from the test run.

    Selection checkboxes for test suite elements

    Selection checkboxes for test suite elements

    Test run configuration

    Run configurations allow to store and manage the current set of test case and smart folder selections for a test run. This enables to quickly change between different test run configurations without the need for manual settings.

    Manage test run configurations

    Manage test run configurations

    Open the run configuration drop-down list
    Click Manage run configurations…
    Add a new run configuration

    Add a new run configuration

    Click the Add button to add a new configuration setting
    Assign the new configuration a meaningful name
    Click OK to confirm and end configuration management

    Result(s):

    • The current run configuration is visible beneath the RUN button in the test suite view
    • Specify test run settings for a run configuration
    • Switch between run configurations to quickly change settings
    Example for different run configurations

    Example for different run configurations

    TestRun configuration including all test suite elements
    TestWithoutValidation configuration excluding the validation smart folder

    Disabling/enabling test suite items

    A test run can also be controlled by specifically disabling/enabling recording modules, code modules, module groups and setup & teardown regions. Disabled modules are not executed during the test run.

    Disabling a test suite item

    Disabling a test suite item

    Select the test suite item which is to be disabled
    Open the context menu and click Disable
    See the corresponding test suite item being disabled
    Note icon

    Note

    • Disabling a module group means, disabling all test suite items within this module group
    • Disabling a setup/teardown region means, disabling all test suite items within this region
    • Selective disabling of individual test suite items is possible

    Run iterations

    There are test requirements where a test case or a smart folder is to be executed iterated times. This section shows how these iterations are defined.

    Specifying run iterations

    Specifying run iterations

    Click F4 to open property pane and change Iteration count

    Result(s):

    • Optional run iterations are displayed within the test suite view
    • In the current example, the test case is iterated 5 times
    Iteration count information in test suite view

    Iteration count information in test suite view

    Auto-retry

    This option lets you automatically re-run failed test cases or smart folders a specified amount of times without aborting test execution. This is useful because, in UI testing, errors will sometimes occur simply because the application under test was unresponsive. In these cases, one solution might be to simply re-run parts of the test.

    Auto-retry function for test suite elements

    Auto-retry function for test suite elements

    Click F4 to open property pane and change Retry count
    Note icon

    Note

    • If there are data bindings or run iterations, the retries will start at the point of failure
    • For example, if the failure occurred at iteration 3 of 5, that’s where the retry will start
    • Only test cases or smart folders that failed every single retry will be marked as failed in the report

    Report levels

    Report levels are a powerful concept for controlling the level of itemization of reports. Managing large and complex test solutions with numerous test cases and possibly thousands of recording and code modules will ask for sophisticated methods to keep test reports as large and detailed as necessary, but as short as possible.

    Further reading

    Report levels are beyond the scope of this chapter, but they are introduced and explained in detail in > Ranorex Studio fundamentals > Reporting > ⇢ Concept of report levels.

    Error behavior

    Test runs, and errors belong together. Designing test solutions with test cases imply the occurrence of errors during their execution. Within the test suite, the behavior of a test run can be controlled based on test cases and smart folders.

    Note icon

    Note

    Control of the error behavior is applicable to test cases and smart folders only.

    Continue with iteration

    Error behavior: Continue with iteration

    Error behavior: Continue with iteration

    Continue with iteration

    •  The test run will be continued with the next iteration of the smart folder ‘Validation’

    Continue with sibling

    Error behavior: Continue with sibling

    Error behavior: Continue with sibling

    Continue with sibling

    • The test run will be continued with the next sibling test case or smart folder, i.e. the smart folder ‘DatabaseCleanUp’

    Continue with parent

    Error behavior: Continue with parent

    Error behavior: Continue with parent

    Continue with parent

    • The test run will be continued with the next parent test case or smart folder, i.e. the test case ‘CompleteDatabaseTest’

    Stop

    • The test run is stopped entirely

    Setting error behavior

    The error behavior is set through the context menu of a test case or a smart folder.

    Specification of error behavior

    Specification of error behavior

    tipp icon

    Hint

    The default error behavior is Continue with sibling.

    Error behavior of setup/teardown regions

    • An error occurred in setup leads to an immediate stop of the test run
    • If one module fails in teardown it continues with the next module in teardown

    Running a test from test suite view

     

    Running a test

    Running a test

    Click the RUN button to start the test
    See the debug information while Ranorex prepares everything for the test
    Watch the progress information during the test
    tipp icon

    Hint

    Keep your fingers from mouse and keyboard during the test. Ranorex performs the test as a user would do it.

    Result(s):

    • Finally, Ranorex Studio opens with the summary of the test report
    Final test report summary

    Final test report summary

    Further reading

    Test reports are introduced and explained in detail in > Ranorex Studio fundamentals > Reporting > ⇢ Introduction.