Test Execution Performance Analysis

Mar 28, 2019 | Best Practices, Product Insights

Performance analysis
When your Test Suite starts to grow and gets more complex, it’s a good idea to take a look at its overall performance and run duration. Identifying time-consuming bottlenecks in your tests has never been easier, as Ranorex Studio 9 comes with powerful analysis tools. Learn how Ranorex managed to tremendously cut down on their daily test run duration by using this new feature.

Fast feedback on new code changes marks one of the fundamentals of agile development processes. If you’re using continuous integration and run daily tests, you’ll want to make sure that results are available on a regular basis. Developers can then investigate where the Test Cases failed and react faster on bugs which might have occurred.

When your products grow more complex and developers add features to the mix, they also affect your Test Suite’s complexity. At a certain point the promise to deliver quick results might turn into a challenge. Sounds familiar to you? You might have reached a point where you should check the performance of your Test Suite.

To ensure constant high product quality, Ranorex runs an automated Test Suite to examine its own software builds every night. The massive Test Suite takes around 12 hours to complete. In order to cut down on this, it’s important to find out which parts are having an impact on the performance. In previous versions of Ranorex, one might have had to use the Report to find out about this, but they were limited in detail. Enter the new Performance Tracing feature in Ranorex Studio 9.

Performance Tracing in Ranorex Studio 9

Performance Tracing gives you all the insights you need to understand which parts of your Test Suite are having an impact at the runtime performance. You will receive detailed data reports on every action that occurred during the test run. You can enable Performance Tracing in the Test Suite settings by selecting a tracer. Ranorex Studio 9 comes with multiple tracers that let you specifically look into an area of interest.

Select a tracer you like to investigate in the Test Suite settings

The feature is disabled by default. You’ll notice that there are tracers for various areas of the test run. If you’re not sure yet in which area you would like to investigate, take the “All” tracer as a starting point. Once you have enabled a tracer and run your Test Suite, Ranorex will generate a CSV file containing a detailed overview of timings for each action. This is not limited to Recording actions, but also provides you insights in the overall workings of your test. This also works when you use Ranorex Remote to execute your test on an agent. Usually, you’ll find the file in the Documents folder on your SUT, but you can specify a different location in the Test Suite settings.

Having a first look on a Repository trace log. A search counter of 7 is a good indicator bearing some RanoreXPath optimization potential.

Working with the Data

When you open a CSV file you will find a log of all actions that Ranorex Studio executed during the test run. Depending on the tracer you’ve selected, the data generated might look different. If you choose the Repo tracer, for example, you’ll get an overview of the RanoreXPaths used and the number of attempts to find the said element.

You will also see the duration, in milliseconds, that it took to complete the action. Obviously, you’ll want to sort the list and have a look at the actions taking the most time. However, you can get much more information from the reports. You can see through the actions to spot delays you might have put during test development and forgot to remove or see unnecessary repetitions. Apart from that, you can use analysis tools like Microsoft Power BI to import the raw data and create dashboard-like reports on your test performance.

Use data analysis tools to get insights like never before
At Ranorex the performance analysis of their daily-run Test Suite revealed a lot of potential for optimizations. Thanks to the insights gained from the performance tracing reports more than 2 hours (almost 20%) of runtime could be conserved. Now it’s your turn to give it a try and optimize your Test Suites!

Related Posts:

5 Software Quality Metrics That Matter

5 Software Quality Metrics That Matter

Which software quality metrics matter most? That’s the question we all need to ask. If your company is dedicated to developing high-quality software, then you need a definition for what “high-quality” actually looks like. This means understanding different aspects of...

The Ins and Outs of Pairwise Testing

The Ins and Outs of Pairwise Testing

Software testing typically involves taking user requirements and stories to create test cases that provide a desired level of coverage. Many of these tests contain a certain level of redundancy. Traditional testing methods can lead to a lot of wasted time and extend...