Lesson 8 - Reporting
By default, each run of a test suite, test case, smart folder, or single module generates a report file (*.rxlog) which tells you if the run was successful or not. In the following lesson you will learn about the following:
Progressive report preview
Since Ranorex 6.1 you can catch the progress of the report of a long test run. So you do not have to wait for the final report at the end of the test and can follow the progress.
Reports are auto-saved during test execution in the report folder and you can open it from there without any worries. By default a report is created in the directory of the test executable. Just open up a file explorer and switch to this directory. From there just open the report file you want to preview. If you have specified your own report directory, you will start the preview from there.
If you open a report which is in progress, an indicator is shown expressing the preview state of the report. To get an update of the report you can manually refresh it or close and re-open the file.
In the following picture you can see how a progressive report is indicated and the refresh button.
Progressive Report Preview indication
The auto save interval for a report is configured in the properties of the test suite. Open up the properties of the current test suite and switch to tab page "Report". In field "Auto Save Interval" you can set the interval in seconds.
In case you execute a test module, which is in a project without a test suite, the auto save interval is 30 seconds by default.
Reading Ranorex Reports
Ranorex report after running a test suite
The report shown after running a test suite provides a general overview of how many of the test containers (test cases, smart folders) have been executed successfully, have failed, or have been blocked. Each executed test container and all its child modules can be analyzed in depth by expanding a particular test container. The pie chart in the upper right only represents the successful, failed, and blocked test cases, not smart folders.
In addition to that, a test suite report also informs you about the following:
- System information like execution time, machine name, operating system, screen dimensions, language, duration, total errors, total warnings
- Global parameter values
- Run iterations and data iterations
Run iterations are indicated by the "Iterations: #" and "Run #" labels next to test containers. "Iterations: #" tells you how many times the test container will be run. Each sub item of the test container represents a single iteration. The number of the iteration is indicated by the "Run #" label next to it.
Data iterations are indicated by the "Rows: #" and "Data Row: #" labels next to test containers. "Rows #" tells you for how many data rows the data container will be iterated. Each sub item of the test container represents an iteration for a single data row. The number of the data row is indicated by the "Data Row #" label next to it.
Detailed view of a report
The detailed log messages generated by a recording or by a code module shown in the figure above consist of a time stamp, a report level, a category, and message text. By default logging is turned on for recordings. Use the Recorder's settings dialog to change the default value and to turn off logging for new recordings. In order to turn logging off or on for a particular action item, open the properties and set the 'Use Default Logging' attribute to false.
Jump to item/test case/smart folder and analyze with Ranorex Spy
Expand the report as shown above and move the mouse pointer over the first log message.
Debug a test run with quick links 'Jump to Item' and 'Open in Spy'
Click at 'Jump to item' to open the module and to select the corresponding action. Use the quick link 'Open in Spy' to analyze the RanoreXPath expression used for the particular action item. This is especially useful in situations when a test run fails with the message 'Failed to find item...' error message. Move the mouse cursor over a test case or smart folder and click on 'Jump to test case/smart folder' to select in the test suite.
Filter Log MessagesUse the checkboxes shown at the top of each module to filter log messages.
Filter for different log levels
For each run of a test suite, a single test case/smart folder or a module, a new log file (*.rxlog) is generated and saved in the project's 'Report' folder. You can open older report files by double-clicking the file in the project view. Specify the file names used for the report files within the test suite's settings dialog.
Log file history within Ranorex Studio project
Note It is not required to run Ranorex Studio to open a report file. You can also open the report file from Windows Explorer. If you're going to copy or send the file by email, it is recommended to use the zipped report as described in Lesson 4: Ranorex Test Suite - Test Suite Settings.
Within Ranorex you are able to use different levels of reporting information importance, from a debug message to a failure information. In addition to predefined report levels, you can also easily define your own levels.
Report Levels describe the importance of a message that is delivered to the report. Using different kinds of report levels can help you keep your reports short and better readable.
Within a test suite or even within a test case/smart folder, you are able to define a report level that describes the minimum level of reporting information (see Lesson 4: Ranorex Test Suite - Test Suite Settings for more information). Report information with a lower level of importance is ignored and does not fill up your report, whereas decreasing a test case's or smart folder's report level could assist you in finding errors in your testing scenario.
Predefined Report Levels
Ranorex provides the following predefined report levels - the level itself is indicated by integer values (which are written in parenthesis):
- Debug (10)
- Info (20)
- Warning (30)
- Error (40)
- Success (110)
- Failure (120)
The first five items describe different levels of importance for report messages. For instance an error message in the report indicates a logical test step error whereas a warning message should only get your attention but not indicate an error.
From report levels 'Debug' to 'Success', the result of the current module is still classified as a success whereas the 'Failure' report level classifies the module result as a failure; this is indicated by a red mark in the report.
Report indicating a failed module
A message indicating the existence of warning messages inside a smart folder
Note A test container's failure depends on the success of its child elements which can either be modules or other test containers (within the rules of the test-suite item hierarchy). A module fails when an exception is thrown or a failure message is being logged. Exceptions are typically thrown by failed validation actions, by searching for nonexistent elements or by an explicit call at code level.
User-defined Report Levels
In addition to predefined report levels, you are able to define your own report level with a custom importance value. You can do that either from code or within the Recorder (see Logging Individual Information for detailed information).
Logging Individual Information
Individual information can be sent to the report either by using usercode (i.e. code modules) or by using the recorder action 'Report'.
Logging Information via Code
You can use one of the following methods from the Ranorex 'Report' class in order to report information at a specific level:
- Ranorex.Report.Debug ("Debug Message");
- Ranorex.Report.Info ("Information Message");
- Ranorex.Report.Warn ("Warning Message");
- Ranorex.Report.Error ("Error Message");
- Ranorex.Report.Success("Success Message");
- Ranorex.Report.Failure("Failure Message");
These methods can be used with a single parameter, the message text. Alternatively they can be used with an additional parameter indicating the category which is set to the default value 'User' in the first overload of such methods. The default category can also be set via code by assigning the 'Ranorex.Report.DefaultCategory' property. The category can be seen in a distinct column in the report. The following code lines demonstrate different reporting methods (and different reporting categories), resulting in the report shown after the code example.
// Reporting information (debug, info) using default Category "User" Ranorex.Report.Debug("This is a Debug Information"); Ranorex.Report.Info("This is a Information"); // Setting Parameter Category to specific value and report a warning and an error Ranorex.Report.Warn("Specific Category","This is a Warning Information"); Ranorex.Report.Error("Specific Category", "This is an Error Information"); // Setting the Default Category Ranorex.Report.DefaultCategory = "My new default category"; // Reporting information (success, failure) using the new default category Ranorex.Report.Success("This is a success information"); Ranorex.Report.Failure("This is a failure Information");
' Reporting information (debug, info) using default Category "User" Ranorex.Report.Debug("This is a Debug Information") Ranorex.Report.Info("This is a Information") ' Setting Parameter Category to specific value and report a warning and an error Ranorex.Report.Warn("Specific Category", "This is a Warning Information") Ranorex.Report.Error("Specific Category", "This is an Error Information") ' Setting the Default Category Ranorex.Report.DefaultCategory = "My new default category" ' Reporting information(success, failure) using the new default category Ranorex.Report.Success("This is a success information") Ranorex.Report.Failure("This is a failure Information")
Report with different reporting methods and categories indicating the different levels of importance
Note Please consider that the report information at the failure level provides a screenshot automatically (if tracing screenshots is enabled). In order to visually retrace the last steps that lead to the current error, screenshots will be provided for the last three actions.
User-defined Report Levels
You are able to define your own report levels with a custom name and a value for the level:
Ranorex.ReportLevel MyNewReportLevel = new ReportLevel("My low Report Level", 25, null); Ranorex.Report.Log (MyNewReportLevel, "This is unimportant information");
Dim MyNewReportLevel As Ranorex.ReportLevel = New ReportLevel("My low Report Level", 25, Nothing) Ranorex.Report.Log(MyNewReportLevel, "This is unimportant information")
Depending on the current Report Level (see next chapter), the report information with the user-defined report level should appear in the report as follows:
Report with information for a user-defined level
Current Report Level
Report information will only appear in the report if its report level is higher than or equal to the current report information level from the parent's test container. The current report level can also be set using code:
Ranorex.Report.CurrentReportLevel = Ranorex.ReportLevel.Parse("Only Highest Importance;90");
Ranorex.Report.CurrentReportLevel = Ranorex.ReportLevel.Parse("Only Highest Importance;90")
This assignment causes the inclusion of reporting information only at a level greater than or equal to 90, which is 'Success' and 'Failure'. The resulting report, which is caused by the same code from above but with a different current report level, can be seen below.
Report with a current report level of 90
Note The current report level can always be overridden by logging information with the report level 'Always'.
Ranorex.Report.Log (Ranorex.ReportLevel.Always, "User", "Any-Case message");
Ranorex.Report.Log (Ranorex.ReportLevel.Always, "User", "Any-Case message")
Screenshots of the current state of the system under test or any UI element can be easily sent to the report.
Screenshot of KeePass toolbar opened directly in Ranorex report
If you call the screenshot method without any arguments, a screenshot of the whole desktop will be available in your report.
Note Screenshots from UI elements can only be made if the element is visible at the time a screenshot is made. Controlling your application using code does not necessarily bring your application to the front and make it visible. To ensure the visibility of UI elements, use the 'EnsureVisible' method which is provided by any element that is derived from Ranorex Adapter class (e.g. 'Text', 'RadioButton', 'Button', etc.). Consequently, rooted folders from repository or even application folders do not ensure visibility. Alternatively you can ensure visibility by using the 'Self' property from that object types (rooted folder or application folder).
Reporting System Summary
The Ranorex reporting class provides a way to simply report a system summary:
System summary provided by Report.SystemSummary()
Logging Information from Recorder
The Ranorex Recorder provides a way to send information directly to the report. Just use the 'Add New Action' button from the toolbar and add a 'LogMessage' Action.
Predefined Report Level
Just like a call from code, you can specify one of the predefined report levels directly within the recorder's actions table.
Custom Report Level
Report messages with a custom report level can also be submitted. Simply type the custom report level into column 'Level' in the action table following this format: name for the custom report level, a semicolon, and the priority as a number.
To use a custom report level named 'Custom' with priority 50 type: Custom;50
Define a report message with a custom level
Screenshots and Snapshots
Since Ranorex Version 3.3, you can send screenshots and even snapshots to your report directly from the recorder's actions table. These two actions require a connection to a repository item in order to specify from which object the screenshot or snapshot should be made.
Usage of the report action using the screenshot and the snapshot action type
Updating the Custom Report Format
Starting with Ranorex 4.0.0 the custom report format has been changed from one single file to a template folder holding all necessary files. After upgrading to Ranorex 4 and opening a solution using custom reports, a warning will appear in the Error pane.
ReportXmlFile is no longer supported
Choose resolve option
Please refer to the online guide on upgrading custom report templates created in versions before Ranorex 7.0.
Create a Custom Report Template
Create a custom report template
A new report template folder has automatically been created at
The newly created files can be shown in project view by pressing the 'Show All Files' button.
Note Press the 'Refresh' button to ensure that the new created folder will be shown.
Show all files in project view
Open the file 'RanorexReport5.css' to edit the style of the report according to your wishes.
Note It's recommended to make customizations at the end of the file.
Open the file 'RanorexReport5.xsl' to edit the structure of the report according to your wishes.
Open the file 'View.rxlog' to get a preview of your customizations using some sample data.