Running some test cases in parallel

Ask general questions here.
simmisj
Posts: 19
Joined: Tue Feb 27, 2018 10:12 am

Running some test cases in parallel

Post by simmisj » Tue Feb 27, 2018 10:16 am

Hi.
I am evaluating Ranorex 8.0.1 in order to understand the functionality it provides.
I have two testcases that do not test any GUI but instead just send a simple email and wait for verification. This is written in a "Code module". There is no reason why these tests could not run in parallel and thus save significant amount of time. Is this not possible and is limited by Ranorex itself?

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Running some test cases in parallel

Post by krstcs » Thu Mar 01, 2018 5:07 pm

Ranorex is designed as a functional User Interface test automation tool. Since a user can only do one thing at a time with the mouse and keyboard, Ranorex is not really setup to run parallel processes for testing.

Having said that, Ranorex is built on .NET which is designed as a multi-threaded framework. So, you could potentially write your own code in a project that does what you want in separate threads. This would require you to manage those threads in your code though.

However, you might be able to use a couple of loops and a well-designed data structure to handle what you are attempting in a non-parallel nature using the .NET async paradigm. For example, if you are just doing API calls, you could loop over each one quickly using asynchronous style programming, then just have the system wait for each of the results. If you do it well, it would look like parallel processing, but it really isn't. It would be faster than straight serial processing, but wouldn't have the overhead of multi-threaded processing. Basically you would send all your requests quickly in one loop, and then wait for the responses in another loop. .NET would take care of storing the responses until you needed them.

My other suggestion would be to run your tests using a Continuous Integration system like Jenkins. You can have multiple Jenkins agents run several tests at a time, one on each system. This would require extra VMs and such for testing though, so it may not be available in your environment.
Shortcuts usually aren't...

simmisj
Posts: 19
Joined: Tue Feb 27, 2018 10:12 am

Re: Running some test cases in parallel

Post by simmisj » Fri Mar 02, 2018 12:14 pm

Thanks for the response.
Yes I could write my own code that handles the multiple threads but that would still only be counted as a single test case in the test suit by Ranorex.
What I am after is for each TestCase to run independently and in parallel with others. As far as I understand this simply is not supported by the framework.

For example in the following picture I have four test cases and TestCase2 does not run until TestCase1 has finished, and so on. Since the code for tests 1 and 2 have no UI interaction and they are totally independent on one another then the ideal situation would be that test cases 1, 2, and 3 would all start at the same time. Test cases 1 and 2 would run in the background using no mouse or keyboard interaction while test case 3 would do UI interaction. Then when test case 3 would finish then test case 4 would start and eventually finish. At some point test cases 1 and 2 would finish and would be included in the report at the end when all tests have completed.
Ranorex4TestCases.PNG
4 test cases in a test suit
Ranorex4TestCases.PNG (9.02 KiB) Viewed 417 times
Is Ranorex so heavily focused on UI tests that it simply does not have this feature and maybe has not even thought about it at all?

P.S. About Jenkins. I am not sure how you can compare Jenkins to Ranorex. We use Team City to build our projects and deploy them.

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 3914
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Running some test cases in parallel

Post by odklizec » Fri Mar 02, 2018 12:31 pm

Hi,

Ranorex is primarily a UI test automation tool and as such does not support parallel running of test cases, because it simply makes no sense to run multiple UI tests in parallel (on the same machine/device). What you are asking for is simply not Ranorex priority. True, Ranorex can do much more than just UI tests, but UI tests are still a primary goal. If you want to run multiple tests in parallel, then you need to run them on a separate machines, using command line tool, from which you can instruct Ranorex to run this or that Test Case. Jenkins is just an example of tool you can use for this purpose. I believe it should be possible to achieve the same goal with Team City? You simply need to start multiple instances of the test suite, on multiple (or the same?) machines, where each instance can run different Test Case. This way you can achieve a semi-parallel run of test cases. The downside of this solution is, that you will still get a number of separate reports, one per each test case started this way.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration