Running testsThis chapter introduces and explains the various settings and configurations to control test runs and explains how to run tests out of the test suite.
Sample solutionThe explanations herein are based on a sample solution which can be downloaded hereinafter.
- Unzip the project directory to any folder on your computer
- Start Ranorex Studio and open the solution file
Test run specificationTest cases and smart folders can be selected to be included in the test run, or deselected to be excluded from the test run. Check test suite elements to include to test run/deselect to exclude them from the test run.
Selection checkboxes for test suite elements
Test run configurationRun configurations allow to store and manage the current set of test case and smart folder selections for a test run. This enables to quickly change between different test run configurations without the need for manual settings.
Manage test run configurations
Add a new run configuration
- The current run configuration is visible beneath the RUN button in the test suite view
- Specify test run settings for a run configuration
- Switch between run configurations to quickly change settings
Example for different run configurations
Disabling/enabling test suite itemsA test run can also be controlled by specifically disabling/enabling recording modules, code modules, module groups and setup & teardown regions. Disabled modules are not executed during the test run.
Disabling a test suite item
- Disabling a module group means, disabling all test suite items within this module group
- Disabling a setup/teardown region means, disabling all test suite items within this region
- Selective disabling of individual test suite items is possible
Run iterationsThere are test requirements where a test case or a smart folder is to be executed iterated times. This section shows how these iterations are defined.
Specifying run iterations
- Optional run iterations are displayed within the test suite view
- In the current example, the test case is iterated 5 times
Iteration count information in test suite view
Auto-retryThis option lets you automatically re-run failed test cases or smart folders a specified amount of times without aborting test execution. This is useful because, in UI testing, errors will sometimes occur simply because the application under test was unresponsive. In these cases, one solution might be to simply re-run parts of the test.
Auto-retry function for test suite elements
- If there are data bindings or run iterations, the retries will start at the point of failure
- For example, if the failure occurred at iteration 3 of 5, that’s where the retry will start
- Only test cases or smart folders that failed every single retry will be marked as failed in the report
Report levelsReport levels are a powerful concept for controlling the level of itemization of reports. Managing large and complex test solutions with numerous test cases and possibly thousands of recording and code modules will ask for sophisticated methods to keep test reports as large and detailed as necessary, but as short as possible.
Error behaviorTest runs, and errors belong together. Designing test solutions with test cases imply the occurrence of errors during their execution. Within the test suite, the behavior of a test run can be controlled based on test cases and smart folders.
Continue with iteration
Error behavior: Continue with iteration
- The test run will be continued with the next iteration of the smart folder ‘Validation’
Continue with sibling
Error behavior: Continue with sibling
- The test run will be continued with the next sibling test case or smart folder, i.e. the smart folder ‘DatabaseCleanUp’
Continue with parent
Error behavior: Continue with parent
- The test run will be continued with the next parent test case or smart folder, i.e. the test case ‘CompleteDatabaseTest’
- The test run is stopped entirely
Setting error behaviorThe error behavior is set through the context menu of a test case or a smart folder.
Specification of error behavior
Error behavior of setup/teardown regions
- An error occurred in setup leads to an immediate stop of the test run
- If one module fails in teardown it continues with the next module in teardown
Running a test from test suite view
Running a test
- Finally, Ranorex Studio opens with the summary of the test report
Final test report summary