Tests randomly fail since update

Ask general questions here.
Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 2:03 pm

Hi there

Since our update to version 7 we've found that certain tests seem to randomly fail.

The message we keep getting is "does not match the specified value (element is not valid)."

When we re-run the test it works fine but this may fail again on a later test execution.

When we rolled back to version 6 the issues stopped occurring.

We've since been adding delays immediately before the validations and we've found (although I can't fully confirm this) that the tests have started passing.

Any thoughts?

Thanks,
Peter

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 2:10 pm

I can now confirm that the delays are not working.

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 3908
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Tests randomly fail since update

Post by odklizec » Tue Aug 15, 2017 2:13 pm

Hi,

I'm afraid, it's pretty hard to suggest something reliable without seeing the app in question, the test itself (at very least screenshot of recording where it typically fails) and exact error message (ideally entire report). But even then, it may be impossible to tell what's wrong.

I would suggest to try update Ranorex with most recent 7.1.1. If the problem still occurs, try to post as much details about the problem as possible. Also, if the issue happens at the same place, add action Report-Snapshot (before/after the failing place), which could help with problem identification.

BTW, if you are using hardcoded delays in your tests, you should definitely replace them with WaitForExists/NotExists actions. These are much more reliable in a long-term run than hardcoded delays.
Last edited by odklizec on Tue Aug 15, 2017 3:15 pm, edited 1 time in total.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 3:06 pm

odklizec wrote:Hi,

I'm afraid, it's pretty hard to suggest something reliable without seeing the app in question, the test itself (at very least screenshot of recording where it typically fails) and exact error message (ideally entire report). But even then, it may be impossible to tell what's wrong.

I would suggest to try update Ranorex with most recent 7.1.1. If the problem still occurs, try to post as much details about the problem a possible. Also, if the issue happens at the same place, add action Report-Snapshot (before/after the failing place), which could help with problem identification.

BTW, if you are using hardcoded delays in your step, you should definitely replace them with WaitForExists/NotExists actions. These are much more reliable in a long-term run than hardcoded delays.
Hi there

I will update to the latest version and failing that I will post back with additional details

Thanks,
Peter

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 3:51 pm

odklizec wrote:Hi,

I'm afraid, it's pretty hard to suggest something reliable without seeing the app in question, the test itself (at very least screenshot of recording where it typically fails) and exact error message (ideally entire report). But even then, it may be impossible to tell what's wrong.

I would suggest to try update Ranorex with most recent 7.1.1. If the problem still occurs, try to post as much details about the problem as possible. Also, if the issue happens at the same place, add action Report-Snapshot (before/after the failing place), which could help with problem identification.

BTW, if you are using hardcoded delays in your tests, you should definitely replace them with WaitForExists/NotExists actions. These are much more reliable in a long-term run than hardcoded delays.
Hi again

I have updated to the latest version 7.1.1 and we're now getting more errors than before (there's a reference to winforms) It's not getting past the first few tests at all now. I've attached a zip file with the errors.
Attachments
Ranorex.zip
(172.49 KiB) Downloaded 25 times

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Tests randomly fail since update

Post by krstcs » Tue Aug 15, 2017 4:25 pm

Have you looked at the release notes? You should read them every time you update to make sure you know the possible issues. In addition there were several API changes in 7.0.

https://www.ranorex.com/release-notes.html
https://www.ranorex.com/release-notes/api-changes.html

Specifically in 7.0:
UserCode methods expect a new alias for the System.Windows.Forms namespace. This can cause compile errors in user code classes after an upgrade complaining about "ambiguous references", e.g. between System.Windows.Forms.Control and Ranorex.Control
-> remove any existing "using System.Windows.Forms;" statement at the top of the file and prefix the ambiguous reference with either "WinForms." (most likely) or "Ranorex." depending on whether you want to reference a System.Windows.Forms or Ranorex type
Shortcuts usually aren't...

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 4:32 pm

krstcs wrote:Have you looked at the release notes? You should read them every time you update to make sure you know the possible issues. In addition there were several API changes in 7.0.

https://www.ranorex.com/release-notes.html
https://www.ranorex.com/release-notes/api-changes.html

Specifically in 7.0:
UserCode methods expect a new alias for the System.Windows.Forms namespace. This can cause compile errors in user code classes after an upgrade complaining about "ambiguous references", e.g. between System.Windows.Forms.Control and Ranorex.Control
-> remove any existing "using System.Windows.Forms;" statement at the top of the file and prefix the ambiguous reference with either "WinForms." (most likely) or "Ranorex." depending on whether you want to reference a System.Windows.Forms or Ranorex type
Hi there

I've read through but I can't find anything related. The part you're referencing is also on a 6.2 release - not 7.
We've updated from 7.0.1 to 7.1.1

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Tue Aug 15, 2017 5:17 pm

In addition - we're now receiving this message which could be the cause of our failures.
Attachments
Capture.PNG
Capture.PNG (40.64 KiB) Viewed 489 times

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Tests randomly fail since update

Post by krstcs » Tue Aug 15, 2017 5:43 pm

Yeah, you're right, I must have missed the transition on the page. Search failure on my part, apologies.


After looking at the text file you attached, it appears that the issue is that there is a problem with object recognition due to the use of .NET 1.1 version of WinForms, is that correct? What .NET framework does your AUT target?

Also, the report data file you attached is corrupted and can't be opened correctly. Please attach one that is complete.

It is possible that the LineageGlueRule exception is the problem, but they usually don't cause issues. They are thrown often on the back-end but don't usually cause tests to fail.

You might want to email support directly at [email protected] (if in North America) or [email protected].
Shortcuts usually aren't...

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Wed Aug 16, 2017 10:19 am

Hi again

Yes I've just sent support an email this morning. Hopefully the problems can be rectified as our test suite is currently unusable. I will post the solutions when available

Peter546
Posts: 20
Joined: Sun Jun 19, 2016 12:23 pm

Re: Tests randomly fail since update

Post by Peter546 » Mon Aug 28, 2017 2:22 pm

I've been in touch with the Ranorex support team. They've analysed our problem and have since raised our issue with their development team.

I will continue to post updates.