Regression testing software can help you make the manual regression testing process automated. It can save a lot of time and effort. But what is regression testing software? What should you look for when you are deciding on the regression testing tool of your choice?...
Ranorex standard reporting
In this chapter, you’ll learn where reports are stored, what they contain, how to read them, and where you can change report settings.
File name and location
The file name of a report is generated automatically. It changes depending on whether the test is executed from a recording module or from a test suite.
- For a run from a test suite, the file name starts with the test suite’s name.
- For a run from a recording, the file name starts with the recording’s name.
- The second part of the report name is a combination of the date and time the report is generated.
- Test date = 21 February 2018 = 20180221
- Test time = 07:08:36 PM = 190836
- Date and time are separated by an underscore.
- The file ending
,short for Ranorex Log
Report file location
Reportsfolder of your project.
The Reports folder in the project’s folder only contains a shortcut to the original reports, and does not contain any of the stylesheet or raw data files.
See here for an illustration:
Reports folder in the output directory of the project.
Layout files for the report.
Shortcuts to the original report files.
Original report files with raw data files.
The original reports along with the layout and raw data files are stored directly in the output directory of your project, and not collected in a Reports folder.
The report header displays a range of useful data and summarizes the test results.
Basic report data
The name of the test suite or recording module from which the test was run.
If there’s at least one warning in the report, an orange notification appears, as shown below.
Error/warning counter in test report
A pie chart summarizing the test results.
A green pie chart means all test cases were successful.
A red pie chart means all test cases failed.
- Pie chart with successful, failed, and blocked test cases
- In the example, the run configuration contained 8 test cases. The first two test cases passed successfully. The third failed, after which the test was aborted, blocking the remaining 5 test cases.
Viewing report details
By default, the details for test cases, smart folders, etc. are collapsed. Simply click the arrow next to an item’s name to show the details.
Basic report structure
The report is structured in the same way as the test suite or recording module on which the test run was based. Each structure element in the test suite appears the report, as well as each test action item. Report messages appear for all actions in the order of execution provided that they met the report level threshold.
Detailed report contents
By default, this column displays the relative execution time measured from the beginning of the test run. This can be changed in the ⇢ report settings.
The second column shows the report level of the action performed.
This column shows the action type performed.
The report message contains more detailed information as to what happened during the execution of the action.
Report message stating that the key sequence ‘Harry’ was performed on a specific repository item.
Data iterations in reports
In data-driven tests, a test case or smart folder bound to data goes through a number of iterations depending on the data source. In the report, each of these iterations is shown separately.
Data iterations in reports
- Data iterations are indicated by the Rows: # labels next to test containers.
- Rows # indicates how many times the data container will be iterated – one time for each row.
- Each data row/iteration and its details are shown in the report.
- Variables and values used in this iteration.
- Secret test data can be masked in the data source dialog. This will also apply to the report.
- If, for example, age and gender are masked, the data appears as shown below:
Masked report data
Run iterations in test reports
If a test case or a smart folder is run iteratively, each run iteration appears in the report with its details.
Run iterations in test reports
- Run iterations are indicated by the Iterations: # and Run: # labels next to test containers.
- Iterations: # tells you how many times the test container is run.
There are two filters available in the report: one for filtering test containers and one for filtering messages for actions.
Filter test containers
Use the checkboxes in the upper part of the report to filter test containers by their success status.
Filtering report messages
Filter action messages
You can also filter action messages in modules by their report level.
Jump to item
You can directly jump from the report message to the corresponding action or test suite item in Ranorex Studio.
Open in Spy
When you open a report outside of Ranorex Studio, the Open in Spy button becomes available. Click the button to open the corresponding report item in Ranorex Spy.
Video reporting lets you record a test run as a collection of videos. This also works on Ranorex Agents.
Enable video reporting
Video reporting is disabled by default. You can enable and configure it in the ⇢ Report settings.
Play video reports from the report
In the report, click Play video next to a test case to play the respective video.
The videos are saved in the output directory in the Reports folder. For each test run, a separate folder is created.
Three video folders
Progressive report preview
Ranorex generates the report as the test run progresses. You can view it at any time during the test run. This is especially useful for very long test runs.
- Ranorex starts generating the report as soon as the test run begins.
- The report file is saved once the defined auto-save time elapses (default = 30s).
- At this point, you can open the in-progress report simply by double-clicking the report file.
Progressive reporting with two notifications indicating that the test is still in progress.
Report settings and configuration
To access the report settings:
Click the Report tab.