Long test runs intermittently fail with no indication why

Bug reports.
gcushman
Posts: 10
Joined: Fri Dec 27, 2013 8:16 pm

Long test runs intermittently fail with no indication why

Post by gcushman » Sat Feb 01, 2014 6:15 pm

I have a regression suite that consists of about 1100 testcases which take 25-26 hours to run using the TestSuite Runner. Ever since upgrading from 4.0.4 to 4.1.4, the suite intermittently fails (approx 10%). The failure is always at different spots and the report viewer doesn't display any information except for the following text:
"Data file (<filename>.rxlog.data) or transformation file is missing."

In Windows explorer I do see a tmp version of the file:
<filename>.rxlog.data_tmp

Examining the file, I can see that it just stopped writing to the file in mid-stream. I have plenty of free disk space on this box (10 gig free). I also checked event viewer around the time of the failure and no errors or warnings to indicate a problem. Also doesn't see any obvious signs of memory leaks and the overall health of the image when this happens appears to be fine.

Is this a known bug in 4.1?
Does Ranorex have a limitation with number of testcases per test suite? I'd prefer to not have to break them up into smaller test runs but if this keeps happening I will have to as it is frustrating to lose 24 hours. Searching through the partial log file for result="Failed" to determine failures combined with re-running the suite at the failure point is also an option but not a very user friendly one.

Any feedback greatly appreciated.
Thanks,
Greg
P.S. I attached screenshot of report viewer as well as zipped folder containing the partial report files.
Attachments
InsightAuto_20140131_054230.rxlog.zip
Zip folder containing report files
(2.53 MiB) Downloaded 118 times
DataFileOrXformationFileMissing_Explorer.png
Screenshot of report files in windows explorer
DataFileOrXformationFileMissing_Explorer.png (14.34 KiB) Viewed 1137 times
DataFileOrXformationFileMissing.png
Screenshot of Report Viewer
DataFileOrXformationFileMissing.png (7.58 KiB) Viewed 1137 times

User avatar
mebner
Posts: 61
Joined: Tue Dec 17, 2013 6:42 pm

Re: Long test runs intermittently fail with no indication why

Post by mebner » Mon Feb 03, 2014 2:18 pm

Hi,

I am not aware of such a bug.
Are you sure that nothing interrupts the test execution?
Do you disable the mouse and the keyboard in your test? If not please disable it since this will ensure that nobody can influence the tests by clicking the mouse or pressing a key. How you can disable it is described here: Disable Mouse/Keyboard Activities.

May I also ask you to enable the internal logging and to send me the created log file of a failed test run?
In order to enable the internal logging for your Ranorex test you just need to copy the attached NLog.config file to the same directory where your test exe resides.

Regards,
Markus

gcushman
Posts: 10
Joined: Fri Dec 27, 2013 8:16 pm

Re: Long test runs intermittently fail with no indication why

Post by gcushman » Mon Feb 03, 2014 9:29 pm

Thanks for the prompt reply. I am sure nobody is touching mouse/keypad and if something like that happened I would expect to see an error such as "unable to find <object>" error and even then it should proceed to the next testcase (the Error Behavior is set to continue to next iteration).

I will send you internal log file when I can capture one.

Thanks,
Greg

gcushman
Posts: 10
Joined: Fri Dec 27, 2013 8:16 pm

Re: Long test runs intermittently fail with no indication why

Post by gcushman » Wed May 06, 2015 4:13 pm

Great news, looks like this is fixed in 5.2.2. I've been running the same 1000+ testcases on an almost nightly basis for the last 2-3 months and have not seen this error since the upgrade.

Thanks Ranorex team!