Ranorex 6.0 performance

Ask general questions here.
Mikhail
Posts: 7
Joined: Wed Jun 22, 2016 10:04 am

Ranorex 6.0 performance

Post by Mikhail » Wed Jun 22, 2016 12:17 pm

Hi,
Currently our automation team use Ranorex Studio version 5.4.6.
We are planning to migrate to version 6.0.
Before migration we decided to make estimation of performance on old version (5.4.6) and new version (6.0).
We installed Ranorex 6, rebuilt our solution and made same operations on both versions.
Several estimation results:

- rebuilding solution:
old version: 4min 02s
new version: 6min52s

- building before running test (time between pressing "Start test case" button and starting execution of test)
old version: 1min 23s
new version: 2min 38s

We have very large solution (about 40 projects in one solution) and we can not send source code of our solution.

The questions are:
1. Is it known issue about low performance with large solution on new version ?

2. We already turned off this option on both versions of ranorex studio:
Tools -> TFS Options -> Enable file state "Modified" check
Is there any other option or settings for increasing performance?

3. Is there any new features on ranorex 6 to organizing work with large solution ?

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Ranorex 6.0 performance

Post by Support Team » Fri Jun 24, 2016 10:23 am

Hello Mikhail,

Currently, there are no known performance issues regarding building large solutions in Ranorex 6.0. I'm afraid there is no special setting to increase building performance either. In order to analyze the issue, we would need the Ranorex Solution. May I ask to contact [email protected]? We could also sign an NDA (Non-disclosure agreement) if it will be required from your side.

We are currently working on improvements concerning "Working in Teams". Which features are part of 6.0 and which features are planned in the future can be found in our product roadmap. All features that were implemented in Ranorex 6.0 can be found in the release notes.

Sincerely,
Johannes
.
Image

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Ranorex 6.0 performance

Post by krstcs » Fri Jun 24, 2016 1:40 pm

In addition to what Johannes said, have you tried using MSBuild directly on your solution from the command-line? Are you sure that it's a Ranorex issue and not an MSBuild or system issue?

Also, are you building your solution with BOTH versions of Ranorex on a VM? Hardware will usually be faster than VMs, so make sure your builds are being run on comparable systems, otherwise the times don't mean much.
Shortcuts usually aren't...

Mikhail
Posts: 7
Joined: Wed Jun 22, 2016 10:04 am

Re: Ranorex 6.0 performance

Post by Mikhail » Fri Jun 24, 2016 4:21 pm

Thanks for the answers.

krstcs, we usually make rebuilding of the solution by native Ranorex build operation.
We did not try any other ways.
I thought it is necessary because Ranorex Studio makes all operation that way (processing of *.rxtst files, generating auto code etc.)

All measurements were performed on one (hardware) machine.


Support Team, how can we send you our solution?
Or I should create a support ticket to discuss all nuances ?

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Ranorex 6.0 performance

Post by krstcs » Fri Jun 24, 2016 4:55 pm

In the background, Ranorex builds your tests on MSBuild, just like any other .NET project, so you can pass the <TestName>.sln file to MSBuild and it will build it just fine, that's the way I do it on my build server. Using MSBuild directly is typically faster than building in Ranorex Studio because Ranorex updates all code files before each build.

However, I was just suggesting this as a way to narrow down the problem. It may or may not make a difference for you.
Shortcuts usually aren't...

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Ranorex 6.0 performance

Post by jasoncleo » Wed Jun 29, 2016 8:55 am

We also are experiencing a serious performance hit after updating to v6.0 of Ranorex. We were previously using Ranorex 5.4.4. Occassionally the Studio IDE doesn't respond for quite a period of time (around the order of 20~30seconds).

The complete build time has shortened (18min on a VM with 5.4.4 has reduced to 15min with 6.0 and debugging enabled).

But building a single project (F9) has become quite slow and this is quite an issue when our team are working on debugging or fine-tuning a test suite.

Previously it would be around 20seconds to build a single project on 5.4.4, but now it is around 50seconds on v6.0.

30seconds doesn't sound like alot extra, but when you are repeatedly making changes, recompiling that project, then all those extra delays add up. I'll be sending a copy of our solution through to support to see if there is something that can be done.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Ranorex 6.0 performance

Post by Support Team » Thu Jun 30, 2016 12:43 pm

Hello Everyone,

We were able to reproduce the performance issues with Jason's solution. The issue is currently analyzed by our development department and I will update this thread as soon as possible.

@Mikhail: Please contact [email protected] in order we can send you the credentials for our ftp server. Alternatively, you could try sending us the solution by WeTransfer.

Sincerely,
Johannes
.
Image

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Ranorex 6.0 performance

Post by Support Team » Thu Jul 07, 2016 9:27 am

Hi Everyone,

We added this issue to our internal bug-tracking system.

Currently, we do not know when this issue will be fixed. This depends on various factors. Nevertheless, all reported bugs and feature requests are documented in the release notes, with every new version release of Ranorex.

Thank you for your understanding.

Sincerely,
Johannes
.
Image

Mikhail
Posts: 7
Joined: Wed Jun 22, 2016 10:04 am

Re: Ranorex 6.0 performance

Post by Mikhail » Mon Oct 10, 2016 5:18 pm

Hello

We are still using Ranorex version 5.4.6.
I keep track changes on release notes. Unfortunately I unable to find any mentions about performance issue, described on this topic.

It would be great if you let know about any updates concerning the issue.

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Ranorex 6.0 performance

Post by krstcs » Mon Oct 10, 2016 7:01 pm

If it isn't in the release notes then it hasn't been fixed yet, since, as Johannes mentioned, they note all fixes to reported issues in the release notes. Until they update the release notes, there is likely no other information that they can/will give because they do not want to say "It will be in the next release" and then not be able to do it because of some unforeseen issue. And, asking for updates from the support team will not speed it up any, as they have no control over if/when something gets addressed.

Generally, they also will comment on the related threads in the forum that the issue has now been fixed, and close the thread, so when that happens it is also a good indicator.
Shortcuts usually aren't...

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Ranorex 6.0 performance

Post by jasoncleo » Tue Oct 11, 2016 2:30 am

I'd forgotten about this thread. We had been experiencing the performance issues with version 6.0.0 and 6.0.1, but we updated in just the past week to 6.1. The performance issue appears to have been resolved.

When we were analysing the behaviour before, it looked like the lag was due to Ranorex Studio parsing all the RXTST files before beginning the build each time. In 6.1, it seems that has been removed and is only done when the solution is opened, or if there are changes made to the solution.

Generally speaking, we are finding 6.1 a big improvement, and also appreciate that support for targeting 4.5.1 and 4.5.2 frameworks has been added. Also it now detects when solution and project files are modified by other processes and offers the option to refresh the solution.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Ranorex 6.0 performance

Post by Support Team » Tue Oct 11, 2016 8:29 am

krstcs wrote:If it isn't in the release notes then it hasn't been fixed yet
Well, this is usually true. We add all fixed issues that are reported by clients to the release notes.
However, it seems we missed this specific performance issue when building single projects. I just added that fix to the release notes for 6.0.2 ("Fixed performance issue in Studio when building single projects or adding a new project") - and the fix is also included in 6.1.0.
Mikhail wrote:It would be great if you let know about any updates concerning the issue.
I'm very positive this also fixes your performance issue - but I can't say for sure unless you give it a try :D

Regards,
Alex
Ranorex Team
.
Image

Mikhail
Posts: 7
Joined: Wed Jun 22, 2016 10:04 am

Re: Ranorex 6.0 performance

Post by Mikhail » Wed Oct 12, 2016 3:32 pm

Thanks to all for reply.
We will try the latest version soon and will give feedback.