Jenkins fails to read Junit xml for publishing Junit Test Results

Bug reports.
sstrand72
Posts: 6
Joined: Thu May 30, 2019 3:38 pm

Jenkins fails to read Junit xml for publishing Junit Test Results

Post by sstrand72 » Tue Jul 16, 2019 9:01 pm

I am using Jenkins for CI, and as such, I am creating a junit xml file after my test suites executes to post results in Jenkins. I have done this within the test suite, as a property AND I have also tried this in the build passing it as a parameter. The junit xml file generates, however, Jenkins fails to read the xml file from time to time because it states the format has trailing content and can not be published. As a result it changes my builds to FAIL or unstable in Jenkins. My tests can all be passing , but because Ranorex is producing/translating a bad xml file, it is causing issues in Jenkins. Has anyone else run into this issue?

I am running Ranorex 9.1.0 and the latest release of Jenkins.

Here is an example of what the output looks in Jenkins:

14:14:57 Object reference not set to an instance of an object.
14:14:59 Execution finished with result Success
14:14:59 Publishing test results
14:14:59 Publishing C:\Users\Stacey.Strand\AppData\Local\Temp\Ranorex\0e8cea6b197816e15504ce44eadca3cc\Report.rxzlog to http://127.0.0.1:8081/api/files/C:/ProgramData/Ranorex Agent/Uploads/files/
14:14:59 Publishing C:\Users\Stacey.Strand\AppData\Local\Temp\Ranorex\0e8cea6b197816e15504ce44eadca3cc\Report.rxlog.junit.xml to http://127.0.0.1:8081/api/files/C:/ProgramData/Ranorex Agent/Uploads/files/
14:14:59 Test results published
14:14:59 --------------------------------------------------------------------------------
14:14:59 | D O W N L O A D R E S U L T S |
14:14:59 --------------------------------------------------------------------------------
14:14:59 Download Report.rxzlog
14:15:00 10%
14:15:00 20%
....
14:15:00 70%
14:15:00 80%
14:15:00 90%
14:15:00 100%
14:15:00 Download Report.rxlog.junit.xml
14:15:00 11%
....

14:15:00 100%
14:15:00 --------------------------------------------------------------------------------
14:15:00 | D O N E |
14:15:00 --------------------------------------------------------------------------------
14:15:00 Job completed: Success
14:15:00 Test execution succeeded
14:15:00 Report: C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxzlog
14:15:00 Report: C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxlog.junit.xml
14:15:00
14:15:00 C:\Jenkins\workspace\NGPWand\M103_TestSuite>exit 0
14:15:00 Archiving artifacts
14:15:02 Recording test results
14:15:02 Build step 'Publish JUnit test result report' changed build result to UNSTABLE
14:15:02 Gitcolony notification failed - java.lang.IllegalArgumentException: Invalid url:
14:15:02 [BFA] Scanning build for known causes...
14:15:02 [BFA] No failure causes found
14:15:02 [BFA] Done. 0s
14:15:02 Disk usage plugin fails during calculation disk usage of this build.
14:15:03 No emails were triggered.
14:15:03 Finished: UNSTABLE


When looking at the xml it has the following information in Jenkins

Stacktrace
Failed to read test report file C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxlog.junit.xml
org.dom4j.DocumentException: Error on line 35207 of document : Content is not allowed in trailing section. Nested exception: Content is not allowed in trailing section.

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 5154
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by odklizec » Wed Jul 17, 2019 7:34 am

Hi,

What kind of Jenkins plugin are you using for publishing JUnit report (standard JUnit plugin)? I'm using JUnint plugin and I'm not experiencing such problems.

At next, are you somehow modifying the report? Could you please share both report (ideally rxzlog) and junit.xml file, so Ranorex folks can examine it? Thanks.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

sstrand72
Posts: 6
Joined: Thu May 30, 2019 3:38 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by sstrand72 » Thu Jul 18, 2019 2:54 pm

I am using the standard junit plugin and I am not altering the report at all in Ranorex (other than the name to "Report", because I want it to be overwritten and not save multiple reports). Attaching my rxzlog and xml for reference (zip file of both). If you look at the xml, it has a bunch of trailing content outside the structure and that has to be done by Ranorex, as it is creating the junit xml.

Note, this does not happen all the time. It appears to happen randomly. I can have successful test suites that encounter this issue, test suites with 1 failure, or multiple failures. The very same job in Jenkins will run once fine and then have several of these issues in a row. It appears there is something in the junit conversion that is getting tripped up. Jenkins just doesn't know how to read or interrupt the xml, so I think it throws up its hands.

Appreciate the help!
Attachments
Report.zip
(2.93 MiB) Downloaded 5 times

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 5154
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by odklizec » Thu Jul 18, 2019 3:06 pm

Hi,

Thanks for the files. I think the problem is caused by corrupted rxlog file? When I tried to open attached rxzlog, it throws some errors. Another errors are thrown during an attempt to extract content of rxzlog. I also noticed, that you are doing PDF conversion at the end of test? I think this could be the source of your problems. I would suggest to eliminate the process of report to PDF conversion from test itself and convert the report (from command line) AFTER the junit file creation.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

sstrand72
Posts: 6
Joined: Thu May 30, 2019 3:38 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by sstrand72 » Fri Jul 19, 2019 1:10 pm

I don't think it is related to the pdf file. We had that problem before we added PDF generation to the test suites. But I often get corrupted rxzlog files that are corrupted from Jenkins runs that I can not open. Is there a possible bug/issue there with corrupted rxzlog files? I never see this problem when I run in Ranorex itself. Only when I run on a remote machine, using Jenkins.

User avatar
RobinHood42
Posts: 306
Joined: Fri Jan 09, 2015 3:24 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by RobinHood42 » Mon Jul 22, 2019 1:08 pm

Hi,

This really sounds like an issue with your Jenkins configuration, e.g., test isn't truly finished, report gets corrupted. Please ensure that the test process isn't forcefully closed in any way. Furthermore, you might have some luck with altering the auto-save interval of your solution -> TestSuite properties -> Report:
TestSuiteProps.png
TestSuiteProps.png (7.33 KiB) Viewed 625 times
Cheers,
Robin :mrgreen:

sstrand72
Posts: 6
Joined: Thu May 30, 2019 3:38 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by sstrand72 » Mon Jul 22, 2019 2:29 pm

Thank you! I am trying that option now - increasing it to 120 seconds to see if that remedies the problem. I am guessing that the longer the report (which ours could have hundreds of test cases) the more we should increase this to allow for a proper save of the log file? I will report back if this cures the problem we are seeing. It may take a few days, as it is not consistently happening. But if it disappears altogether, we will know we struck gold!

raghu.pagidi
Posts: 1
Joined: Tue Jul 30, 2019 4:27 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by raghu.pagidi » Tue Jul 30, 2019 5:10 pm

sstrand72 wrote:
Mon Jul 22, 2019 2:29 pm
Thank you! I am trying that option now - increasing it to 120 seconds to see if that remedies the problem. I am guessing that the longer the report (which ours could have hundreds of test cases) the more we should increase this to allow for a proper save of the log file? I will report back if this cures the problem we are seeing. It may take a few days, as it is not consistently happening. But if it disappears altogether, we will know we struck gold!
Did you try with 120 seconds and got positive results on it

sstrand72
Posts: 6
Joined: Thu May 30, 2019 3:38 pm

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Post by sstrand72 » Wed Jul 31, 2019 2:25 pm

Here is the follow up after several overnight runs in Jenkins:

- The increase to 120 seconds did not solve the problem. But I suspect we will run into this eventually as our test reports will grow and grow.
- Here is what I changed that DID seem to solve the problem.
In Jenkins, I was renaming the Report file to something like "Report", instead of just leaving to whatever setting I had in my Ranorex job. I did this by a command override parameter in my Jenkins job. I also set the flag in the Test Suite properties to let Ranorex do the junit file and NOT have Jenkins do it.

The combination of those seemed to do the trick. Thanks for the advice!