Remote execution on TFS Test Agent erratic

[missmarple]

Hi all,

I have noticed that when I execute tests remotely using TFS 2012, Test Agent version 11.0.50727.1 and Ranorex 4.1.4, about 50% of test executions fail (!) because objects are not recognised.
In this particular case, I tested against an HTML5 page on IE 8, using the Ranorex-recommended Coded UI wrapper for Ranorex. All the "connective glue" works, the test starts, etc. - however, there seems to be a real issue with object recognition - the tests stop at various different steps.
I don't think Ranorex is at fault here, since "standalone" execution on the same machine is 100% reliable.
I am running these tests on a corporate network with (unfortunately...) plenty of restrictive Windows policies. Also, the remote machines with the test agents installed are virtual machines with comparatively little CPU power assigned to them - probably doesn't help either.
Has anyone come across something similar ? Before going into too much detail, perhaps someone can comment ? I have plenty of details available if needed...

Regards,
MM

[Support Team]

Hi missmarple,

Since the test run fails “only” 50% of the time, we should be able to exclude a security related issue. Assumed that the permissions didn’t change. The processing power of the machine could be much more problematically, if the search timeout was defined too stingy. Please try to raise the search timeout. Please provide more information concerning your test if the issue still occurs.

Regards,
Robert

[missmarple]

Hi Robert,

thanks for getting back to this. I don't think processing power is the problem in this particular case - although the virtual machines we're using are fairly crappy in terms of performance.
The element that fails to get recognised (consistently, actually - so this might be something we can sink our teeth into) is a drop-down box and its contents that is limited by what is already typed into the text field.
It's a kind of text field/drop down combo thingy. Local exeution 'delivers' every time, whereas remote execution fails every time on this step.
What info do you need from me to proceed here: detailed class/element info, screenshots, debug stuff ?
Let me know.

Thanks in advance,
miss marple

[testautomator]

One possibility could be visibility of the object or resolution. Have you added 'if visible click'? Also, sometimes on VMs the click will not be consistent since speed of mouse and clicks matter.

[missmarple]

Hi testautomator,

thanks for the tip; the dropdown-list entry is actually visible after the previous step has completely (and without playback problems) entered the item to look for. Ranorex just doesn't seem to be able to find it. I think my next angle of attack is going to be a more robust/flexible/defensive object recognition method. I can't remember which attribute I chose, I'll have a play around with others to see if that makes any difference. I'll report back if I make any progress.

Thanks,
mm

[testautomator]

Sure. Let us know please. We will also learn with you

[missmarple]

Well - looks like a schoolboy error to me
The remote machine had IE8 installed (yes, I know - don't ask =) and as soon as I had installed IE10, not only did the problem go away, rather, Ranorex just blasted through the automation. So, in this instance: my mistake. Will test on Chrome just to see.
In a nutshell: no object recognition re-work necessary, MS Test Agents work just fine, just need to update your infrastructure !

Regards,
mm

[krstcs]

Just remember that upgrading infrastructure is not always possible, depending on the type of testing and the target environment.

I test my company's eCommerce website and since it is customer facing, I have to use whatever the customers will use, so IE 8/9/10/11 all must be tested as well as some versions of FF, Chrome, and Safari.

If this is an issue with Ranorex and IE8, then it probably still needs to be addressed by Ranorex so that those of us who don't have the option of not using IE8 aren't stuck.

This also means it probably wasn't an error on your part, just a miss understanding of the way IE 8 presents HTML vs. how IE 9/10/11 do it. I have issues with that as well, and I have had to find ways around it with very careful manipulation of the RXPaths for my objects.