Lesson 4: Ranorex Test Suite

As was already mentioned in Lesson 2: Ranorex Modules - Test Actions, it is recommended to separate a test case into reusable and parameterizable test automation modules. Within the Ranorex test suite you can not only easily create new test cases by combining already existing automation modules - you can also specify global parameters and data connectors to set up data-driven test cases.

Within the following lesson you will learn about:

Test Suite Editor

After the creation of a new test suite project, a Ranorex Studio project automatically contains a <ProjectName>.rxtst file.

By default the test suite file is opened automatically.

: Ranorex Test Suite file

Using the project created during the lessons 1 through 3 you can now create a new test case based on the existing modules. To open up the test suite, you can double-click the *.rxtst file or use the 'View Test Suite' button shown in the toolbar panel.

: Ranorex Studio tool bar

Add a New Test Case

You can add a new test case by clicking the 'NEW' button. Depending on the current item selection of the test suite you are allowed to add different types of test suite elements. A test suite can be made up of the following items:

Test cases
Recording modules
Code modules
Module groups
Folders

Select the 'Test Case' item to create a new test case at the current position.

: Add a new test case

Note: Within a test suite, test cases can be added at different levels. A test case can also be added as a child to an existing test case or to a folder item.

The newly created test case will be used for validating the correctness of KeePass' version number so the name of the test case should be changed. Select and click on the test case and rename it to 'TC_ValidateVersionNumber'.

: Rename the test case

Reuse of Existing Modules

Now you are ready to reuse two existing record modules from the Ranorex Module Browser View.

You can simply drag and drop items from the module browser into the test case or you can add new or existing items using the context menu as shown.

: Add an existing recording module

: Select 'StartSUT' module from the list

Press 'OK' to add the record module to the test case.

Repeat the steps to add the record items 'LogIn' and 'CloseSUT' to the test case.

Insert a New Recording Module

For now the test case only covers starting the application under test, logging in and closing the application under test using three existing modules.

In order to validate the version number of the KeePass application, you can create a new recording module to be used in the test case.

Use the context menu again, but this time to insert a new recording into the test case.

: Insert a new recording module

Before starting the new recording, you should confirm that KeePass is running - you could start the application manually if it is not running.

The new recording module 'ValidateVersionNumber' should cover the following steps:

open the about dialog
validate the version number
close the about dialog

Keep in mind that you need to select the 'Global Recording' option because the system under test will be started by the reused recording module 'StartSUT'.

: Newly created recording scenario

During the validation step, Ranorex automatically created a new repository item for the cell holding the version number from KeePass - the text itself is used for identification of the cell (path). Since the content of the cell (KeePass' version number) is used for identification, a higher version number would cause the test case to fail because the cell can't be found anymore - independent from the match of version number in the validation step itself. To avoid this, the path to this cell should be modified to be more robust. The path can be edited using the RanoreXPath Editor. Additionally it should not use the version number itself for identification.

: Click magic wand to open Advanced RanoreXPath Editor

: Change attributes for identification: deselect 'text'-attribute and select 'index'-attribute

Using the index for identification assures the accessibility of the cell even if a new version is released. To complete this recording you could modify the validation action to use a new variable called 'varCurrentKeePassVersion' with a default value '2.20.1' instead of matching against the constant value. This module variable could be bound to a new global variable 'GlobalCurrentKeePassVersion'.

Now the test case 'ValidateVersionNumber' is ready to be executed.

Use the context menu item 'Play Selected Test Case' to run it and see whether it works or not.

: Play selected test case

Module Group Editor

It can be useful to combine modules which belong together to groups to reuse them more easily.

One scenario is to combine the modules 'StartSUT' and 'LogIn' into a module group 'StartAndLogin' within the KeePass sample project as these two modules are needed to start the system under test. The same applies to the modules 'Save' and 'CloseSUT' which can be combined with the module group 'SaveAndClose'.

This can be done using the module group editor.

In addition to the test suite file, the test suite project also contains a file defining the module groups existing in the project. This module group file is generated automatically as <Project-Name>.rxtmg.

Double-clicking this file in the project view will open it in the module group editor.

: Open module group editor

After creating a new solution an empty module groups file is created.

: Module group editor

You can add module groups to your project in the following ways:

Press the 'New Module Group' button in the module group editor
Use the context menu in the test suite editor to add a module group to a test case
Use the context menu in the test suite editor to group selected modules into a module group

: Adding a module group to a test case using the context menu in the test suite editor

: Grouping existing modules into a new module group using the context menu in the test suite editor

By grouping existing modules into a module group, the module group will be filled with the selected modules. When adding a new module group, the module group is empty and can be filled with modules by using the context menu in the module group and then adding a new or existing module.

: Adding an existing module to a module group using the context menu

An existing module can also be added using drag and drop from the module browser or project view into the module group in the module group editor.

: Drag and drop module into a module group

To allow for a data driven approach as described in Lesson 3 - Data-Driven Testing, it is necessary to pass values from data sources or parameters through the module group to the modules contained in that module group.

You can define data binding for modules in the module group by using the context menu item 'Data Binding...' which appears by right-clicking a module group.

: Open the data binding dialog for module group

The data binding dialog allows you to specify which of the module variables should be changeable from outside the module group and which of the module variables should have constant values inside the module group.

Note: The mechanism for using constant values within module groups allows hiding module variables. This can reduce the complexity and increases the clarity of module groups because it is not always necessary to set each module variable individually.

: Define which module variables should get their values passed from outside the module group and which module variables should have constant value

After defining a module group and its data binding, it can be added to a test case the same way a module is added.

The data binding for module groups works exactly the same way as it works for modules.

Further details about working with modules and data binding can be found in Lesson 2: Ranorex Modules - Test Actions and Lesson 3: Data-Driven Testing.

: Adding a module group to a test case

General Structure of a Test Suite and its Test Cases

In the last section you learned how to add new test cases to a test suite by combining existing modules with a newly created recording.

To see the different ways of organizing a more complex test suite project, open the sample test suite project 'KeePassTestSuite' from the Ranorex Studio start page.

: Opening the sample test suite project 'KeePassTestSuite'

This sample includes different types of elements which can be used within a test suite and covers all possible combinations.

: Ranorex test suite structure

#1 Folder	Used to group multiple test cases
#2 Test case	Represents a test case which can contain modules, a setup or teardown region or other test cases
#3 Setup region	Groups modules used to prepare a test case (e.g. start system under test, initialize a database, etc.)
#4 Teardown region	Groups modules used to clean up a test case (e.g. deleting files generated during test execution, closing application, etc.)
#5 Module group	Used to group several modules into a reusable set
#6 Code module	Automation module written in code
#7 Recording module	Automation module generated by recording

Setup and Teardown Regions

The setup and teardown regions are used to prepare and clean up a single test case execution.

The setup region will be executed before any other module held by the test case and should hold any modules needed to bring the system under test in exactly the state your test can start from. A typical application of this section is to start the application under test and log in with a user.

The teardown region will be executed when the execution of the test case has been finished, which means after all modules have been executed, or an error has aborted the test case. The teardown region should hold any modules needed to clean up the system under test and bring it to the original state. A typical application of this section is to delete all added data and close the application under test.

The setup region will automatically be placed at the beginning and the teardown region will automatically be placed at the end of a test case.

Use the context menu to specify which modules or module groups from a test case should be part of the setup or teardown process.

: Add module to a setup region

Note: If you want to define one setup and one teardown region for a set of test cases, simply nest your test cases as shown in the following picture

: Nested test case for a general setup and teardown region

In order to quickly deactivate a certain module, instead of deleting it from the test case, use the context menu item 'Disable'.

Searching For Elements

Use the 'Search' text box to find elements in the Test Suite.

: Search result for text 'Add'

Running a Test Suite

To run a test suite click the 'RUN' button shown in the test suite editor's toolbar. Use the check boxes to specify whether single test cases or a group of test cases should be part of a test suite run. Create different test suite run configurations using the drop-down box from the toolbar.

These test suite run configurations define if a specific test case will be executed or not.

For example one test suite run configuration can be defined where the data driven tests stored in your test suite will be executed and one test suite run configuration can be defined where they won't.

: Activate different test suite run configurations

: Add or remove test suite run configurations

Running Tests without Ranorex Studio

As you already learned in Lesson 1: Getting Started, Ranorex Studio creates an executable file from your test suite project. In order to execute the test suite in a runtime environment, you have to have the generated executable (*.EXE) and the test suite file (*.RXTST) in the same directory. If your Ranorex Studio solution consists of more than one project, you need to ensure that the library (*.DLL) files are also part of the same directory. In short, to deploy a Ranorex test to a runtime machine it's required to copy the complete output folder (e.g. 'bin/debug') to the target machine.

You can execute the test suite outside Ranorex Studio using one of the following:

Ranorex Test Suite Runner

Simply double-click the *.RXTST file from the project's output folder to open the Ranorex Test Suite Runner.

: External Ranorex Test Suite Runner

You can use the Ranorex Test Suite Runner to run the test suite, execute certain test cases or just run a specific automation module.

Additionally one can create new run configurations the same way as is done within a Ranorex Studio Project.

Running Tests via Command Line

Using the following, you can execute the test suite from the command line:

<GeneratedTestSuite>.exe /<argument>

Allowed arguments:

help|?

Prints this help text.

listconfigparams|lcp

Lists all settable configuration parameters and their values.

config|cfg:<config parameter name>=<value>

Set values for configuration parameters.

reportfile|rf:<report file path>

Sets the name (and path) of the report file. If no path is provided, the current directory is used. By default, the filename specified in the rxtst file is used (for example: %S_%Y%M%D_%T.rxlog).

zipreport|zr

Compresses the report (including associated files) to a single archive ('.rxzlog' extension).

zipreportfile|zrf:<zip report file path>

When used with /zipreport|zr, sets the name (and path) of the compressed report file. If no path is provided, the path of the report file is used. If the file extension is not '.rxzlog', the extension will be replaced with '.rxzlog'. By default, the report filename specified in the rxtst file or the value of reportfile|rf is used with an '.rxzlog' extension (for example: %S_%Y%M%D_%T.rxzlog).

reportlevel|rl: Debug|Info|Warn|Error|Success|Failure|<any integer>

Sets the minimum report level that log messages need to have in order to be included in the log file. Specify 'None' to completely disable reporting.

These levels correspond to the following integer values:

Debug=10
Info=20
Warn=30
Error=40
Success=110
Failure=120

listglobalparams|lp

Lists all global parameters and their values.

testcase|tc:<name or guid of test case>

Runs this test case only.

testsuite|ts:<path to test suite file>

Runs the test cases defined by the test suite (rxtst) file.
By default

the rxtst file with the same name as the <TestSuiteExe> is used
or the first rxtst file in the same folder as <TestSuiteExe>.

runconfig|rc:<configuration name>

Runs the test cases of the specified configuration defined by the rxtst file. Configurations can be edited using Ranorex Studio or TestSuiteRunner. By default, the currently selected run config is used.

module|mo:<module name or guid>

Runs the module with the specified name or guid. Assemblies loaded by <TestSuiteExe> and assemblies referenced in the rxtst file are searched.

param|pa:<global parameter name>=<value>

Creates or overrides values for global parameters specified in the test suite.

runlabel|rul:<custom value>

Sets a custom runlabel for the test run.

Note: Compressed report files (*.rxzlog) can be extracted by right-clicking them in explorer and choosing 'Extract' from context menu. Report files (*.rxlog) can be compressed by right-clicking them in explorer and choosing 'Compress' from context menu.

Some Examples for using Command Line Arguments

Using the KeePass sample test suite - included in Ranorex installation - as reference, the following samples will fulfill the following use cases:

Start the test suite with setting a global parameter:

KeePassTestSuite.exe /pa:GlobalExecutionPath=.\KeePass\KeePass.exe

Start only a specific test case of the test suite:

KeePassTestSuite.exe /tc:TC_OpenLoginAndClose

Start the test suite with generating a zipped report called report.rxzlog:

KeePassTestSuite.exe /zr /zrf:report.rxzlog

Start a specific run configuration of the test suite:

KeePassTestSuite.exe /rc:SmokeTest

Test Suite Settings

Open the test suite's settings dialog by selecting the context menu item 'Properties' on the test suite root node.

: General settings of a test suite

: Global parameters table

General Test Suite Settings

Name	Specifies the name of the test suite (same as shown within the test suite editor)
Description	Description of the test suite (same as shown within the description column of the test suite editor)
Show Progress Dialog	Specifies whether a progress dialog should be shown during test suite execution or not
Capture Tracing Screenshots	Specifies whether tracing screenshots should be captured or not in the case of an error
Report Level	Specifies the level of report shown with the report file

Additional Report Settings

Report Enabled	Specifies whether a test report should be enabled or not
Report File Directory	Specifies the directory for the generated report files
Report File	Specifies the filename generated by the report; move the mouse over the 'Description' label to see the variable values
Compressed Copy	Specifies whether the report should be generated as compressed folder or not
Report Template Folder	Specifies the directory holding customized style files used instead of the Ranorex default style to present the reports; a new template can be created by pressing the button 'Create Custom Template' as explained in 'Lesson 8: Reporting - Create a Custom Report Template'
Auto Save Interval	Specifies how often the report file is saved during an execution
Place screenshots in folder	Specifies whether the created screenshots will be stored in a specific folder for each report or directly in the report folder

Global Parameters

Globally specified parameters can be accessed and used by every test case within the test suite. The KeePassTestSuite example project uses a global parameter to specify the directory path for the application under test. You can easily connect global parameters with variables as shown in the test case 'TC_AddEntry'. In addition you can use the global parameter to transfer values or states between test cases.

Test Case Settings

The 'General' tab of the test case's properties dialog is mainly used to setup how a broken test case impacts other test cases within the test suite.

Name	Name of the test case
Description	Specifies the filename generated by the report; move the mouse over the 'Description' label to see the variable values
Report Level	Specifies the level of messages written to the report file
Error Behavior	Specifies the behavior of the test case and the test suite in case of on error. For further information please have a look at the next paragraph.

The following figures try to illustrate several Error Behavior settings:

Set to 'Continue with iteration' if, in case of an error, the case should not break continue with the next iteration
Set to 'Continue with sibling' if, in case of an error, the currently running test case should break and the execution should continue with the sibling test case
Set to 'Continue with parent' if, in case of an error, the currently running test case should break and the execution should continue with the sibling of the parent test case
Set to 'Stop' in order to break the test suite run in case of an error

: Continue with Iteration

: Continue with Sibling

: Continue with Parent

: Stop

: Test case properties dialog

Use the context menu to change the report level or the error behavior for multiple test cases.

: Specifying error behavior for multiple test cases

Using Data in Test Suites

As already explained in Lesson 3: Data-Driven Testing you can make your automation data-driven by using variables in your modules (recordings, code modules) as well as in your repositories.

These variables can be connected to internal or external data sources and parameters.

Different kinds of Data Container

Let's have a look at the different kinds of data container:

Module Variables:

Variables are the interface between a module and the nesting test case, or - if module groups are used - the module group, respectively.

Variables can be used in

recoding modules (as described in section Using Variables within Recordings),
code modules (as described in section Using Variables with Code Modules) and
repositories (as described in section Using Variables within the Repository).

After building up the test suite with its test cases as described in the previous sections, the included module variables can be bound to group variables, constant values, data columns, or parameters.

Constant Values:

Constant values can be used in module groups to hide module variables outside a module group. This can help to reduce the complexity and increases the clarity of module groups.

For further details have a look at the section about the Module Group Editor.

Group Variables:

In module groups, group variables are the interface between the nested modules and the nesting test case.

For further details have a look at the section about the Module Group Editor.

Data Columns:

The columns of a data connector are called Data Columns and specify the columns from external data sources.

Have a look at the data connectors section (Data Connectors) to get an overview about the different kinds of data connectors. Data columns can be connected to variables in the data binding dialog of a test case as described in section Combining Variables with Test Data.

Parameters:

Module variables can also be bound to parameters. A distinction is made between local parameters, which are part of a specific test case and are available in the test case and all child test cases, and global parameters which are available in the whole test suite.

Unlike local parameters, global parameters can be set from the command line as explained in section Running Tests via Command Line.

Have a look at the section Combining Variables with Parameters to see how parameters can be connected to variables.

Scope of Data Container

Global parameters are available in the whole test suite. This means you can bind global parameters in any test case of a test suite.

Local parameters and data columns will be inherited by all children of a test case. This means you can bind all local parameters and data columns of a test case in its child test cases.

The following figure will illustrate the scope of different data containers.

: Scope of data containers

The green rectangle illustrates the scope of global parameters, the blue rectangle the scope of local parameters and data columns of the test case named 'Parent_Test_Case_1' and the red rectangle the scope of local parameters and data columns of the test case named 'Test_Case_1'.

This means, in 'Test_Case_1' you can bind all global parameters as well as all local parameters and data columns of itself and its parent test case 'Parent_Test_Case_1', as illustrated in the following figure.

: Data binding dialog of Test_Case_1

The green rectangle illustrates a global parameter, the blue rectangle a local parameter and a data columns of the test case named 'Parent_Test_Case_1' and the red rectangle a local parameter and data column of the test case named 'Test_Case_1'.

Modify Parameter and Data Column Values

Parameter values as well as data column values can be changed by a module. In order to transfer values from one module to another, parameter or data column values are updated from their bound module variables once the module has finished execution.

This means you can freely transfer values within the scope of a parameter or data column.

One example of this application would be to dynamically get a value from the UI which should be reused in another module as described in the section Lesson 5: Ranorex Recorder - Additional Editing Options - Types of Action Items addressing the Get Value action or at Code Examples - Accessing Test Case & Test Suite Context.