Ranorex in the cloud - Azure

Experiences, small talk, and other automation gossip.
User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Ranorex in the cloud - Azure

Post by jasoncleo » Wed May 11, 2016 4:54 am

Hi all,

Has anyone had any experience or tried running their Ranorex runtimes on cloud servers?

Some background:
In the past, we've always used VMs on our local ESX servers to spool up and host our Ranorex runtime instances. As they were all on our intranet and local subnet, it was easy to have them connect to network fileshares and move files around, as well as get their floating licenses from our local Ranorex license manager. The ESX server would hold up ok for a modest number of test VMs running at the same time.

Current challenge:
We're now working on a large project that involves developing (and of course testing) a cloud based application, so they've chosen Azure. While we could continue to use our old VMs on our local ESX to run the Ranorex tests, there was the thought that since we now have an Azure account, we could leverage some basic VMs from Azure to have our tests run there. They would theoretically be less prone to performance issues on a solid cloud infrastructure, rather than our dated ESX server.

So what am I actually asking?
I've managed to work out techniques to programmatically start/stop the Azure VMs, as well as open up the RDPs to them. But the problems I'm trying to work out are:
  • 1. Due to the virtual nature of cloud VMs (they present as a single DNS, but with unique ports), will that cause issues with the Ranorex license server? (Our IT hasn't opened ports yet for my trial Azure VMs to ping our license server for me to trial this yet)
  • 2. Have people managed to work out how to easily get data to and from Azure VMs? I'm stuck on that one at the moment.
Also just in general, has anyone tried this before and is it worth all the headache, or should we just stick with our local VM network?

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Ranorex in the cloud - Azure

Post by jasoncleo » Thu Nov 24, 2016 8:37 am

Our company reached the stage where we thought we'd push ahead and so this week I've started with trying to put both our Ranorex license manager and all our automated test environments into Azure.

We're still working our way through the migration, so I'll be updating this thread as we learn about what works and what doesn't. Hopefully it'll be helpful for other people looking to do a similar thing.

Ranorex License Server
We created a Windows Azure VM with 2 cores and 2GB of RAM, but SSD drive. This server will actually be hosting a number of support services for the team, hence it was given more CPU power than necessary.

Because we want this license server to be accessible from both other Azure VMs and our on-premise network; we assigned it a static IP, and configured the firewall within the host to allow inbound TCP/UDP port 7266. We also set up the NSG for this resource group to also allow that inbound rule as well.

To check connection, you can uses SysInternals psping to make sure we could ping port 7266 on the new Azure VM. If you are getting connection refused, then check with your IT group to see if your corporate firewall (not the Azure firewall) is preventing you from reaching that port.

As a note: I found that most people recommend that the license server be made available outside Azure via VPN to your corporate network, rather than just being exposed to the internet like I've done above. This is something I'm still yet to look into.

More to come:
While I've already created a number of additional Azure VMs to serve as test environments and installed Ranorex Studio on those, I still need to finish configuration for our CI environment to be able to deploy Ranorex binaries, and remotely execute them on each of the VMs, then copy back the rxlogs.

At the moment I'm working with the idea of using a Resource Storage file share which all the tests can dump their results into, and then be a single point for our CI to copy back from, using the Azure API.

Once I have a working solution, I'll post about that here.

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Ranorex in the cloud - Azure

Post by jasoncleo » Thu Apr 20, 2017 6:16 am

Wow, the time has really slid by on this one. I had promised to come back to it once it was working, but things had been flat out, so here I am 5 months later.

Where did it get to?
We got it working with our CI and CD solutions. To achieve a robust implementation, we rolled our own .Net solution that leveraged the Microsoft Azure client libraries which were able to control the Azure test VMs, and access the Azure file storage.

We also created our own 'agent' which sat as a service on each Azure test VM and did remote operations. This just kept a TCP connection open so that the controlling utility on the central VM could trigger operations such as starting a Ranorex test, checking whether the test was still running, etc.

Azure allows a neat function for having file storage mapped as a network drive on the VMs, so once we uploaded our test Ranorex binaries, each Azure VM could then access those files to create their own copy for local execution, as well as copy back report files that we could download for our own test reporting metrics.

We managed to achieve a working implementation, and had it running for about a month, where CI was building our Ranorex tests, uploading the binaries, running them distributed across the Azure VMs, and pulling back the results for processing / records.

So is it recommended?
No.

After a month, we ended up canning the whole thing and migrating the test environment back to on-premise VMs. So the CI still exists, but the tests are run on VMs that are hosted on our own ESX infrastructure.

The reasons why the whole thing was canned are:
  • Azure is damn expensive. We were powering the Azure VMs on and off, but this was for CI, so the VMs were on most of the time during business hours, and we had longer soak tests over the weekend. VMs are billed per hour (or part-thereof).
  • Azure is slow. We had a decent amount of binaries to upload (120 test suites which for us equated to 220MB and 2000 files), and then after a test run we had to download all the reports and the associated screenshots which could be in the order of 1GB and more that 10,000 files.
  • The execution on the VMs were fine, as we specced them as reasonable machines and they run off SSD drives which compensates the lower RAM and CPU cores, but they couldn't be easily leveraged by other testers for debugging due to the hassle of getting binaries to them
  • Finally our particular AUT had 3rd party system dependencies. We couldn't get static IPs for all our test VMs as we had too many for our subscription level, and the 3rd parties were not going to open their firewall to the whole internet.
Conclusion:
Bottom line, for the hassle and lost time in data transfers, just tell your infrastructure team to keep the test environment on-premise.

It's technically possible to have your test environment hosted in the cloud, but in our experience there was no benefit from it when we had to manage the entire thing ourselves.

User avatar
Stub
Posts: 173
Joined: Fri Jul 15, 2016 1:35 pm

Re: Ranorex in the cloud - Azure

Post by Stub » Thu Apr 20, 2017 8:21 am

Canned. Whoa :shock: Very interesting story though, jasoncleo, I liked reading the 3 acts including the finale. :)

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 3922
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Ranorex in the cloud - Azure

Post by odklizec » Thu Apr 20, 2017 12:50 pm

Interesting and very useful article! Thanks for sharing your experience!
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Ranorex in the cloud - Azure

Post by krstcs » Thu Apr 20, 2017 12:52 pm

Yeah, this is interesting to me as we've looked into something like this but with AWS instead of Azure. Thanks for the great write-up Jason!
Shortcuts usually aren't...

nmathew
Certified Professional
Certified Professional
Posts: 8
Joined: Thu Nov 06, 2014 6:40 pm
Contact:

Re: Ranorex in the cloud - Azure

Post by nmathew » Fri Aug 04, 2017 5:52 pm

Thank you Jason for sharing your experience