Lesson 4 - Ranorex Test Suite
As was already mentioned in Lesson 2: Ranorex Modules - Test Actions, it is recommended to separate a test case into reusable and parameterizable test automation modules. Within the Ranorex test suite you can not only easily create new test cases by combining already existing automation modules - you can also specify global parameters and data connectors to set up data-driven test cases.
Test Suite Editor
By default the test suite file is opened automatically.
Ranorex Test Suite file
Ranorex Studio tool bar
Adding a new test case or smart folderYou can add a new test case or smart folder by clicking the 'ADD' button. Depending on the current item selection of the test suite you are allowed to add different types of test suite elements. A test suite can be made up of the following items:
- Test cases
- Smart folders
- Recording modules
- Code modules
- Module groups
These items can be added and arranged according to a special hierarchy. You can find out more about it here.
Add a new test case
The newly created test case will be used for validating the correctness of KeePass' version number so the name of the test case should be changed. Select and click on the test case and rename it to 'TC_ValidateVersionNumber'.
Rename the test case
Reuse of Existing Modules
Now you are ready to reuse two existing record modules from the Ranorex Module Browser View.
You can simply drag and drop items from the module browser into the test case or you can add new or existing items using the context menu as shown.
Add an existing recording module
Select 'StartSUT' module from the list
Press 'OK' to add the record module to the test case.
Repeat the steps to add the record items 'LogIn' and 'CloseSUT' to the test case.
Insert a New Recording Module
For now the test case only covers starting the application under test, logging in and closing the application under test using three existing modules.
In order to validate the version number of the KeePass application, you can create a new recording module to be used in the test case.
Use the context menu again, but this time to insert a new recording into the test case.
Insert a new recording module
Before starting the new recording, you should confirm that KeePass is running - you could start the application manually if it is not running.
The new recording module 'ValidateVersionNumber' should cover the following steps:
- open the about dialog
- validate the version number
- close the about dialog
Keep in mind that you need to select the 'Instant Recording' option because the system under test will be started by the reused recording module 'StartSUT'.
Newly created recording scenario
During the validation step, Ranorex automatically created a new repository item for the cell holding the version number from KeePass - the text itself is used for identification of the cell (path). Since the content of the cell (KeePass' version number) is used for identification, a higher version number would cause the test case to fail because the cell can't be found anymore - independent from the match of version number in the validation step itself. To avoid this, the path to this cell should be modified to be more robust. The path can be edited using the path editor. Additionally it should not use the version number itself for identification.
Click the 'Edit in Spy' button open the path editor
Change attributes for identification: deselect text attribute and select column- and row-index attributes
Now the test case 'ValidateVersionNumber' is ready to be run.
Use the context menu item 'Run Selected Test Case' to run it and see whether it works or not.
Run selected test case
Module Group Editor
For example combining the modules 'StartSUT' and 'LogIn' into a module group 'StartAndLogin' within the KeePass sample project could be useful, because these two modules are needed to start the system under test. The same applies to the modules 'Save' and 'CloseSUT' which can be combined with the module group 'SaveAndClose'.
This can be done using the module group editor.
In addition to the test suite file, the test suite project also contains a file defining the module groups existing in the project. This module group file is generated automatically as <Project-Name>.rxtmg.
Double-clicking this file in the project view will open it in the module group editor.
Open module group editor
Module group editor
- Press the 'New Module Group' button in the module group editor
- Use the context menu in the test suite editor to add a module group to a test case/smart folder
- Use the context menu in the test suite editor to group selected modules into a module group
Adding a module group to a test case using the context menu in the test suite editor
Grouping existing modules into a new module group using the context menu in the test suite editor
Adding an existing module to a module group using the context menu
Next to adding existing modules it is possible to add folders to module groups to add additional hierarchical levels.
An existing module can also be added using drag and drop from the module browser or project view into the module group in the module group editor.
Drag and drop module into a module group
You can define data binding for modules in the module group by using the context menu item 'Data Binding...' which appears by right-clicking a module group.
Open the data binding dialog for module group
The data binding dialog allows you to specify which of the module variables should be changeable from outside the module group and which of the module variables should have constant values inside the module group.
Note The mechanism for using constant values within module groups allows hiding module variables. This can reduce the complexity and increases the clarity of module groups because it is not always necessary to set each module variable individually.
Define which module variables should get their values passed from outside the module group and which module variables should have constant value
Note By pressing the button 'Auto-Create' a group variable for each unbound module variable will be generated based on the name of the module variable. By pressing the button 'Auto-Bind' unbound module variables will be bound to group variables having identical names.
The data binding for module groups works exactly the same way as it works for modules.
Further details about working with modules and data binding can be found in Lesson 2: Ranorex modules and test actions and .
Adding a module group to a test case
General structure of a test suite
To see the different ways of organizing a more complex test suite project, open the sample test suite project 'KeePassTestSuite' from the Ranorex Studio start page.
Opening the sample test suite project 'KeePassTestSuite'
This sample includes different types of elements which can be used within a test suite and covers all possible combinations.
Ranorex test suite structure
|#1 Test suite||Represents the entire test suite and is the root level of the structure|
|#2 Test case||Represents a test case|
#3 Setup region
|Groups modules used to prepare a test case (e.g. start system under test, initialize a database, etc.)|
|#4 Smart folder||Used to organize other items. Smart folders can be nested (a smart folder may contain other smart folders)|
|#5 Module group||Used to group several modules into a reusable set|
#6 Teardown region
|Groups modules used to clean up a test case (e.g. deleting files generated during test execution, closing application, etc.)|
#7 Code module
|Automation module written in code|
|#8 Recording module||Automation module generated by recording|
These items follow a hierarchy. This means that there are rules that define which items may be added directly to other items in the test suite structure. The table below illustrates this.
Test suite: At the root level, you may add test cases, smart folders, and setup/teardown regions.
Test cases: You may add smart folders, setup/teardown regions, modules and module groups.
Smart folders: You may add test cases*, other smart folders, setup/teardown regions, modules** and module groups**.
Setup/teardown: You may add modules and module groups.
*Only if the smart folder is not part of a test case.
**Only if the smart folder is part of a test case.
Note If a drag/drop or copy/paste operation would create a nested test case, the affected test case will be converted into a smart folder instead. The resulting smart folder will work the same way as the original test case.
Setup and teardown regions
The setup and teardown regions are used to prepare and clean up a single test case/smart folder run.
The setup region will be run before any other module held by the test case/smart folder and should hold any modules needed to bring the system under test in exactly the state your test can start from. A typical case for using this section is to start the application under test and log in with a user.
The teardown region will be run after all the modules in the test case/smart folder or when an error causes the test case/smart folder run to abort. The teardown region should hold any modules needed to clean up the system under test and bring it to the original state. A typical case for using this section is to delete all added data and close the application under test.
The setup region will automatically be placed at the beginning and the teardown region will automatically be placed at the end of a test suite, test case, or smart folder.
Use the context menu to specify which modules or module groups from a test case or smart folder should be part of the setup or teardown process.
Add module to setup region
Note If you want to define one setup and one teardown region for a set of test cases, simply use a smart folder as shown in the following picture
Smart folder containing test cases for a general setup and teardown region
Note In order to define global setup and teardown regions click on 'Add Setup' or 'Add Teardown' in context menu of the test suite node and drag and drop the wanted modules or module groups to displayed the setup and teardown regions.
Show setup and teardown region at the test suite level
Drag and drop modules to global setup region
In order to quickly deactivate a certain module, instead of deleting it, use the context menu item 'Disable'.
Searching for elements
Use the 'Search' text box to find elements in the test suite.
Search result for text 'Add'
Running a Test Suite
To run a test suite click the 'RUN' button shown in the test suite editor's toolbar or the run button in the Ranorex Studio toolbar. This will always run the test suite using the currently selected run configuration.
Run configurations define which test cases and smart folders will be executed during a test. Use the check boxes to specify which elements you want to be run. Ticking the checkbox at the test suite level will tick all others. You can also save your current configuration. Here's how:
- Check the boxes of the desired elements.
- Click the drop-down menu in the test suite editor's toolbar and select "Manage Run Configurations...".
- Click "Add".
- Enter a name and click OK.
- You can now apply the run configuration by selecting it from the drop-down menu.
Note Run configurations can only be used with the test suite they were created in.
Activate different test suite run configurations
Add or remove test suite run configurations
Run iterations let you run specific test cases or smart folders any amount of times during a test run. You can enable them as follows:
- Click on the test cases(s) or smart folder(s) you want to iterate.
- Go to "View" and click on "Properties" or press F4. The properties pad will appear.
- Enter your desired number of iterations in "Iteration Count".
The selected items will now be run the specified amount of times during test execution.
This option lets you automatically rerun failed test cases or smart folders a specified amount of times without interrupting test execution. This is useful because in UI testing, errors will sometimes occur simply because the AUT was unresponsive. In these cases, the solution is to simply rerun the test.
To activate this option, follow these steps:
- Select a test case or smart folder
- Press F4 to bring up its properties.
- Enter the desired number of retries in the ‘Retry Count’ field.
If there are data bindings or run iterations, the retries will start at the point of failure. For example, if the failure occurred at iteration 3 of 5, that’s where the retry will start. Only test cases or smart folders that failed every single retry will be marked as failed in the report.
Running Tests without Ranorex Studio
As you already learned in Lesson 1 - Getting Started, Ranorex Studio creates an executable file from your test suite project. In order to execute the test suite in a runtime environment, you have to have the generated executable (*.EXE) and the test suite file (*.RXTST) in the same directory. If your Ranorex Studio solution consists of more than one project, you need to ensure that the library (*.DLL) files are also part of the same directory. In short, to deploy a Ranorex test to a runtime machine it's required to copy the complete output folder (e.g. 'bin/debug') to the target machine.
You can execute the test suite outside Ranorex Studio using one of the following:
- Ranorex Test Suite Runner
- Command Line
Ranorex Test Suite Runner
Simply double-click the *.RXTST file from the project's output folder to open the Ranorex Test Suite Runner.
External Ranorex Test Suite Runner
Additionally one can create new run configurations the same way as is done within a Ranorex Studio Project.
Running Tests via Command Line
Note All of the following examples are based on the desktop and iOS sample projects that are included in Ranorex Studio.
Using the following command pattern, you can execute the test suite from the command line:
To use one of the available arguments, use this command pattern:
To use more than one argument, separate the arguments with a space, like so:
<GeneratedTestSuite>.exe /<first argument> /<second argument>
KeePassTestSuite.exe /zr /ju
Note When executing a test suite from command line the return value '0' signals the successful execution of the test script, the return value '-1' signals a failure, and the return value '42' signals no available license.
The table below lists all available command line arguments. Each argument has a full and a short version. You can use either as you like and even mix them.
Replace any text in angle brackets <> with the respective values of your project.
|Full argument||Short version||Description|
Shows the help text including all available arguments.
Lists all settable configuration parameters and their values.
|config:<config parameter name>=<value>||cfg:<config parameter name>=<value>||
Set values for configuration parameters.
|endpoint:<endpoint name>||ep:<endpoint name>||
Display name of the endpoint to be used as the automation root during test execution. If no endpoint is specified 'localhost' is used.
Please note: The solution ships with no endpoints. The endpoint iPad was choosen for demonstration purposes.
|reportfile:<report file path>||rf:<report file path>||
Sets the name (and path) of the report file. If no path is provided, the current directory is used. By default, the filename specified in the rxtst file is used. (for example: %S_%Y%M%D_%T.rxlog).
Compresses the report (including associated files) to a single archive (".rxzlog" extension).
Creates the report in JUnit format as well. The report will be placed in the same location as the original Ranorex Report.
|zipreportfile:<zip report file path>||zrf:<zip report file path>||
When used with argument zipreport, sets the name (and path) of the compressed report file. If no path is provided, the path of the report file is used. If the file extension is not ".rxzlog", the extension will be replaced with ".rxzlog". By default, the report filename specified in the rxtst file or the value of argument reportfile is used with an ".rxzlog" extension (for example: %S_%Y%M%D_%T.rxzlog).
|reportlevel:<report level>||rl:<report level>||
Sets the minimum report level that log messages need to have in order to be included in the log file. Possible values are: None, Debug, Info, Warn, Error, Success, Failure, or any integer value. Specify 'None' to completely disable reporting. These levels correspond to the following integer values: Debug=10, Info=20, Warn=30, Error=40, Success=110, Failure=120
Lists all global parameters and their values.
|listtestcaseparams:<name or guid of test case or smart folder>||ltcpa:<name or guid of test case or smart folder>||
Lists all test case or smart folder parameters and their values.
|testcase:<name or guid of test case or smart folder>||tc:<name or guid of test case or smart folder>||
Runs this test case or smart folder only.
|testsuite:<path to test suite file>||ts:<path to test suite file>||
Runs the test cases or smart folders defined by the test suite (rxtst) file.
Please note: The solution ships with only one test suite. The test suite file 'AlternativeKeePassTestSuite' was choosen for demonstration purposes.
|runconfig:<run configuration name>||rc:<run configuration Name>||
Runs the test cases or smart folder of the specified run configuration. Run configurations can be edited using Ranorex Studio or TestSuiteRunner. By default, the currently selected run configuration is used.
|module:<name or guid of module>||mo:<name or guid of module>||
Runs the module with the specified name or guid. Assemblies loaded by <GeneratedTestSuite> and assemblies referenced in the rxtst file are searched.
|param:<global parameter name>=<value>||pa:<global parameter name>=<value>||
Sets or overrides values for global parameters.
|testcaseparam:<name or guid of test case or smart folder>:<parameter name>=<value>||tcpa:<name or guid of test case or smart folder>:<parameter name>=<value>||
Sets or overrides values for test case or smart folder parameters.
Please not: The parameter for test case AddEntryWithArguments does not exist in the solution and is just to explain the usage above.
|runlabel:<custom value>||rul:<custom value>||
Sets a custom runlabel for the test run.
|testcasedatarange:<name or guid of test case or smart folder>=<data range>||tcdr:<name or guid of test case or smart folder>=<data range>||
Sets the data range for a test case or smart Folder.
Note Compressed report files (*.rxzlog) can be extracted by right-clicking them in explorer and choosing 'Extract' from context menu. Report files (*.rxlog) can be compressed by right-clicking them in explorer and choosing 'Compress' from context menu.
Test Suite Settings
General settings of a test suite
Global parameters table
General Test Suite Settings
|Name||Specifies the name of the test suite (same as shown within the test suite editor)|
|Description||Description of the test suite (same as shown within the description column of the test suite editor). Use the HTML text editor to format the description and to add links to the description. The formatted description will be represented as HTML content in the generated report|
Show Progress Dialog
|Specifies whether a progress dialog should be shown during test suite execution|
|Report Level||Specifies the minimum report level of messages shown with the report file|
|Warn for unbound variables||Specifies whether a warning should be shown for unbound variables|
Globally specified parameters can be accessed and used by every test case within the test suite. The KeePassTestSuite example project uses a global parameter to specify the directory path for the application under test. You can easily connect global parameters with variables as shown in the test case 'TC_AddEntry'. In addition you can use the global parameter to transfer values or states between test cases.
Additional Report Settings
|Enable Report||Specifies whether a test report should be generated|
|Report File Directory||Specifies the directory for the generated report files|
|Report File||Specifies the filename generated by the report; the following placeholders are available:
%L Run Label
%C Run Config Name
%H Host Name
%S Test Suite Name
%X Test Suite Result
|Auto Save Interval||Specifies how often the report file is saved during execution|
|Timestamp||Specifies whether the timestamp should be computed relative to the test suite start time or relative to the test module start time or the machine time should be taken|
|Compressed Copy||Specifies whether a copy of the report should be generated as compressed folder|
|JUnit-compatible file||Specifies whether a JUnit-compatible file should be generated in addition to the test report.|
|Collect screenshots in a subdirectory||Specifies whether the created screenshots will be stored in a specific folder for each report or directly in the report folder|
|Show invisible characters in values||Specifies whether invisible characters such as spaces or tabs will be shown in values in report messages|
|Report Template||Specifies the directory holding customized style files used instead of the Ranorex default style to present the reports; a new template can be created by pressing the button 'Create Custom Template' as explained in|
|Tracing Screenshot Mode||Specifies whether the screenshots will be captured in foreground (before action is executed) or background (while action is being executed) or capturing tracing screenshots is disabled|
|Quality||Specifies the image quality of the captured screenshots|
Test Case and smart folder settings
The 'General' tab of the test case's or smart folder's properties dialog is mainly used to setup how a broken test case or smart folder impacts other test cases or smart folders within the test suite.
|Name||Name of the test case/smart folder|
|Description||Description of the test case/smart folder (same as shown within the description column of the test suite editor). Use the HTML text editor to format the description and to add links to the description. The formatted description will be represented as HTML content in the generated report|
|Report Level||Specifies the level of messages written to the report file|
Specifies the behavior of the test case/smart folder and the test suite in case of an error. For further information please have a look at the next paragraph.
The following figures try to illustrate several Error Behavior settings:
Continue with iteration: Ranorex will continue with the next iteration of the current test case or smart folder.
Continue with sibling: Ranorex will stop executing the current test case or smart folder and continue with the next sibling test case or smart folder instead.
Continue with parent: Ranorex will stop executing the current test case or smart folder and continue with the next parent test case or smart folder instead.
Stop: Ranorex will abort the test entirely.
Test case/smart folder properties dialog
Specifying error behavior for multiple test cases
Using Data in Test Suites
These variables can be connected to internal or external data sources and parameters.
Different kinds of Data Container
Variables are the interface between a module and the containing test case or smart folder, or - if module groups are used - the module group, respectively.
Variables can be used in
Constant values can be used in module groups to hide module variables outside a module group. This can help to reduce the complexity and increases the clarity of module groups.
For further details have a look at the section about the Module Group Editor.
In module groups, group variables are the interface between the nested modules and the nesting test case.
For further details have a look at the section about the Module Group Editor.
The columns of a data connector are called Data Columns and specify the columns from external data sources.
Have a look at the data connectors section (Data Connectors) to get an overview about the different kinds of data connectors. Data columns can be connected to variables in the data binding dialog of a test case as described in section Combining Variables with Test Data.
Module variables can also be bound to parameters. A distinction is made between local parameters, which are part of a specific test case or smart folder and are available in the test case/smart folder and all child smart folders, and global parameters which are available in the whole test suite.
Unlike local parameters, global parameters can be set from the command line as explained in section Running Tests via Command Line.
Have a look at the section Combining Variables with Parameters to see how parameters can be connected to variables.
Scope of data containers
Global parameters are available in the whole test suite. This means you can bind global parameters to any module in any test case or smart folder of a test suite.
Local parameters and data columns of a test case or smart folder are inherited by all child test cases or smart folders. This means you can bind them to
- any modules contained in the parent test case or smart folder.
- any modules contained in child test cases or smart folders.
The following figure illustrates the scope of different data containers.
Scope of data containers
Green rectangle: Scope of global parameters.
Blue rectangle: Scope of local parameters and data columns of SmartFolder2.
Red rectangle: Scope of local parameters and data columns of TestCase2.
This means that you can bind the following parameters and data columns to the modules in TestCase2:
all global parameters (green rectangle)
all local parameters and data columns of SmartFolder2 (blue rectangle)
all local parameters and data columns of TestCase2 (red rectangle)
This is illustrated in the following screenshot:
Data binding dialog of TestCase2
Modify Parameter and Data Column Values
Parameter values as well as data column values can be changed by a module. In order to transfer values from one module to another, parameter or data column values are updated from their bound module variables once the module has finished execution.
This means you can freely transfer values within the scope of a parameter or data column.
One example of this application would be to dynamically get a value from the UI which should be reused in another module as described in the section Lesson 5: Ranorex Recorder - Additional Editing Options - Types of Action Items addressing the Get Value action or at Code Examples - Accessing Test Case & Test Suite Context.
Conditions allow you to control whether a test container will be run during test execution based on specific rules that you can define. You can set up one condition per test case or smart folder, but a condition can have up to 10 rules.
To set up a condition, right-click on a test container in the test suite and click ‘Condition…’
The condition window will open.
If the condition doesn’t contain any rules, it is in its empty state and will look like this:
Click on ‘Add rule’ to add the first rule to the condition. The window will change to look like this:
|#1||This is the entire condition for the test container.|
|#2||Click this checkbox to activate or deactivate the condition. Only active conditions will be applied during test execution.|
|#3||If you have added multiple rules to the condition, specify whether all or any number of them needs to be matched for the test container to be run. If the condition contains only one rule, you can ignore this selection.|
|#4||A rule, not yet complete. See below for a more detailed description of rules.|
|#5||Click here to clear all rules from the condition and return it to its empty state.|
|#6||Click here to clear this rule from the condition.|
|#7||Click here to add another rule to the condition. You can add up to 10 rules to a condition.|
Rules tell Ranorex what to check to determine whether the condition as a whole has been fulfilled. Rules can apply to data sources or parameter sources. Depending on which of the two the rule applies to, the selections you can make will change. Once you’ve fully configured a rule, it is considered complete. This is indicated by a blue icon next to the rule. If the rule is incomplete, an orange icon will appear in the missing fields and next to the rule.
A complete rule, as indicated by the blue icon
An incomplete rule, as indicated by the orange icons
Note When a rule is complete, it only means that all fields contain valid values. It does not mean that the rule will be matched during test execution.
Configuring a rule to apply to a data source
- Select Data source.
- Ranorex will auto-select the data source of the test container. If no data source has been assigned to the test container, the rule cannot be completed.
- Select the column that contains the value you want the rule to check.
- Select the operator.
- Select an existing value taken from the column you selected in 3. or enter your own value.
Configuring a rule to apply to a parameter
- Select Parameter.
- Select whether the parameter is a global or local one.
- Select the parameter whose value you want the rule to check.
- Select the operator.
- Select an existing value taken from the parameter you selected in 3. or enter your own value.
Once you’ve added all the rules you need, click on ‘Apply’ or ‘OK’ to save the condition. Clicking ‘Cancel’ will return the condition to its last saved state and discard your changes.
The state of the condition will then be indicated by an icon in the test suite view. You can double-click this icon to directly access the condition window.
with one or more rules