Page 1 of 1

Generating Tests to Use in Stress Testing

Posted: Tue Jun 07, 2016 10:30 pm
by karltinsly
I have read the other threads and responses concerning stress testing. I know that Ranorex is a functional testing tool, not a load testing tool, but let me explain what we're trying to accomplish.

We are a large electric cooperative, and we are looking for a way to simulate a large outage situation. To do this, we would like to simulate a large amount of outages being entered into our system through all the various avenues that can happen. In our case, this means customer service reps entering outages through our CIS system (a Java Swing gui-based system), members entering outages through our customer care website (web-based, we support IE, Safari, Chrome, and Firefox), and through our IVR (we would probably have to code this part separately).

When it came time to run our simulation, we would like to have - for example - fifty users in the CIS and 100 members on the website entering outages at a rate of 100 every ten minutes, which would continue for an hour or two. Each of these simulated users would running off of a record set, since each outage would need to be entered on a unique account and service location.

We don't need to measure performance and loads like a lot of load testing tools would do - our network and system admins can do that in real time during the test. We just need to simulate all those users.

I know that what I've described is not what Ranorex is designed for, but my question is, could Ranorex be used to create the individual record-driven tests that could then be called by another program to simulate all those users? If so, does anyone have any suggestions for a program to run and control all those individual tests?

Thanks for any insight!

Re: Generating Tests to Use in Stress Testing

Posted: Wed Jun 08, 2016 1:09 pm
by odklizec
Hi,

In my opinion, doing what you are after with Ranorex is a waste of time and energy ;) The main problem you will face is how to simultaneously run multiple instances of Ranorex exe to simulate simultaneous entering of data by multiple users. In a theory, you can achieve this by a continuous integration tool like Jenkins and a stack of virtual machines.

The thing is, that each instance of exe will have to be run on a separate desktop/VM. You cannot simultaneously run two or more instances of Ranorex exe on the same machine. Running two Ranorex tests at the same time would be as if two physical users would try to use mouse and keyboard at the same time;) In a theory, you can use separate user accounts (on the same machine), but then each user have to be logged-in and his desktop unlocked.

And at second, even if you run each exe on a separate VM, you will soon run out of available Ranroex licenses. You see, each exe started on a separate VM will consume one license. So if you run for example 10 instances of exe on 10 separate VMs at the same time, you will need 10 Ranorex runtime licenses available. You see the problem?

I would suggest to check the HP LoadRunner, which does exactly what you want and HP offers a free version, with 50 virtual users, which could be run at the same time. I've examined it and successfully created a demo test with it. While it's nowhere as elegant and easy to adopt/use as Ranorex, it does its job very well with simulating multiple users at the same time. Hope this helps?

Re: Generating Tests to Use in Stress Testing

Posted: Wed Jun 08, 2016 6:28 pm
by karltinsly
Thanks for your reply, Pavel. So tests generated by Ranorex can only be run in Ranorex? I like Ranorex because it is easy to record tests. I'm not a coder, and it's appealing to be able to create usable tests myself.

I was hoping that I could use the tests created by Ranorex outside of Ranorex. In other words, record a single test of entering an outage through our CIS, then use another tool like loadrunner to run that test in multiple VMs, but with different record sets.

Do you think that could be done?

Thanks again.

Re: Generating Tests to Use in Stress Testing

Posted: Wed Jun 08, 2016 7:31 pm
by krstcs
As Pavel said, that could be done, functionally. Ranorex can do anything .NET can do.

Also, Ranorex tests are compiled .NET executables, so they can run anywhere that .NET is installed, but they require the Ranorex libraries to always be on the system as well.


There are several issues with what you want though:

1. You still have to have a distinct license for EACH CONCURRENTLY RUNNING INSTANCE of Ranorex or a Ranorex test. So, in your case you would need 100-150 licenses (you could probably use runtime licenses on all of the remote systems). There may be a way for Ranorex to work out a site license for a large number of runtime licenses, but you would need to discuss that with their sales folks ([email protected]).

2. Ranorex can only run a single instance of a test at a time on a system because it requires sole use of the mouse/keyboard, so you would need 100-150 systems (desktops or more probably VMs) to run each individual Ranorex test instance.

3. You would need to either install Ranorex on each remote system or you would need to copy the correct library files over to each system (this can probably be done through a Continuous Integration (CI) solution like Jenkins, Bamboo, or Team City).


As Pavel said, you should consider using a real load test system instead as they are designed to do this very thing, where Ranorex would require quite a bit of extra work and cost on your part.

I love Ranorex (the tool AND the team!!) and would love to see them integrate something like this in the future, but you have to use the right tool for the job, and I don't think that is Ranorex in this case at this time.

Re: Generating Tests to Use in Stress Testing

Posted: Thu Jun 09, 2016 2:42 pm
by Support Team
Hello guys,

Thank you very much for your input. I'm happy to say that we are already investigating integration for performance testing tools. Unfortunately, we cannot release any official status updates at this time, but we will be happy to discuss this offline. Please email us directly at ([email protected]) to discuss this further.

I look forward to hearing from you.

Regards,
Bernhard