Page 1 of 1

Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Tue Jul 16, 2019 9:01 pm
by sstrand72
I am using Jenkins for CI, and as such, I am creating a junit xml file after my test suites executes to post results in Jenkins. I have done this within the test suite, as a property AND I have also tried this in the build passing it as a parameter. The junit xml file generates, however, Jenkins fails to read the xml file from time to time because it states the format has trailing content and can not be published. As a result it changes my builds to FAIL or unstable in Jenkins. My tests can all be passing , but because Ranorex is producing/translating a bad xml file, it is causing issues in Jenkins. Has anyone else run into this issue?

I am running Ranorex 9.1.0 and the latest release of Jenkins.

Here is an example of what the output looks in Jenkins:

14:14:57 Object reference not set to an instance of an object.
14:14:59 Execution finished with result Success
14:14:59 Publishing test results
14:14:59 Publishing C:\Users\Stacey.Strand\AppData\Local\Temp\Ranorex\0e8cea6b197816e15504ce44eadca3cc\Report.rxzlog to http://127.0.0.1:8081/api/files/C:/ProgramData/Ranorex Agent/Uploads/files/
14:14:59 Publishing C:\Users\Stacey.Strand\AppData\Local\Temp\Ranorex\0e8cea6b197816e15504ce44eadca3cc\Report.rxlog.junit.xml to http://127.0.0.1:8081/api/files/C:/ProgramData/Ranorex Agent/Uploads/files/
14:14:59 Test results published
14:14:59 --------------------------------------------------------------------------------
14:14:59 | D O W N L O A D R E S U L T S |
14:14:59 --------------------------------------------------------------------------------
14:14:59 Download Report.rxzlog
14:15:00 10%
14:15:00 20%
....
14:15:00 70%
14:15:00 80%
14:15:00 90%
14:15:00 100%
14:15:00 Download Report.rxlog.junit.xml
14:15:00 11%
....

14:15:00 100%
14:15:00 --------------------------------------------------------------------------------
14:15:00 | D O N E |
14:15:00 --------------------------------------------------------------------------------
14:15:00 Job completed: Success
14:15:00 Test execution succeeded
14:15:00 Report: C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxzlog
14:15:00 Report: C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxlog.junit.xml
14:15:00
14:15:00 C:\Jenkins\workspace\NGPWand\M103_TestSuite>exit 0
14:15:00 Archiving artifacts
14:15:02 Recording test results
14:15:02 Build step 'Publish JUnit test result report' changed build result to UNSTABLE
14:15:02 Gitcolony notification failed - java.lang.IllegalArgumentException: Invalid url:
14:15:02 [BFA] Scanning build for known causes...
14:15:02 [BFA] No failure causes found
14:15:02 [BFA] Done. 0s
14:15:02 Disk usage plugin fails during calculation disk usage of this build.
14:15:03 No emails were triggered.
14:15:03 Finished: UNSTABLE


When looking at the xml it has the following information in Jenkins

Stacktrace
Failed to read test report file C:\Jenkins\workspace\NGPWand\M103_TestSuite\Report.rxlog.junit.xml
org.dom4j.DocumentException: Error on line 35207 of document : Content is not allowed in trailing section. Nested exception: Content is not allowed in trailing section.

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Wed Jul 17, 2019 7:34 am
by odklizec
Hi,

What kind of Jenkins plugin are you using for publishing JUnit report (standard JUnit plugin)? I'm using JUnint plugin and I'm not experiencing such problems.

At next, are you somehow modifying the report? Could you please share both report (ideally rxzlog) and junit.xml file, so Ranorex folks can examine it? Thanks.

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Thu Jul 18, 2019 2:54 pm
by sstrand72
I am using the standard junit plugin and I am not altering the report at all in Ranorex (other than the name to "Report", because I want it to be overwritten and not save multiple reports). Attaching my rxzlog and xml for reference (zip file of both). If you look at the xml, it has a bunch of trailing content outside the structure and that has to be done by Ranorex, as it is creating the junit xml.

Note, this does not happen all the time. It appears to happen randomly. I can have successful test suites that encounter this issue, test suites with 1 failure, or multiple failures. The very same job in Jenkins will run once fine and then have several of these issues in a row. It appears there is something in the junit conversion that is getting tripped up. Jenkins just doesn't know how to read or interrupt the xml, so I think it throws up its hands.

Appreciate the help!

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Thu Jul 18, 2019 3:06 pm
by odklizec
Hi,

Thanks for the files. I think the problem is caused by corrupted rxlog file? When I tried to open attached rxzlog, it throws some errors. Another errors are thrown during an attempt to extract content of rxzlog. I also noticed, that you are doing PDF conversion at the end of test? I think this could be the source of your problems. I would suggest to eliminate the process of report to PDF conversion from test itself and convert the report (from command line) AFTER the junit file creation.

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Fri Jul 19, 2019 1:10 pm
by sstrand72
I don't think it is related to the pdf file. We had that problem before we added PDF generation to the test suites. But I often get corrupted rxzlog files that are corrupted from Jenkins runs that I can not open. Is there a possible bug/issue there with corrupted rxzlog files? I never see this problem when I run in Ranorex itself. Only when I run on a remote machine, using Jenkins.

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Mon Jul 22, 2019 1:08 pm
by RobinHood42
Hi,

This really sounds like an issue with your Jenkins configuration, e.g., test isn't truly finished, report gets corrupted. Please ensure that the test process isn't forcefully closed in any way. Furthermore, you might have some luck with altering the auto-save interval of your solution -> TestSuite properties -> Report:
TestSuiteProps.png
Cheers,
Robin :mrgreen:

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Mon Jul 22, 2019 2:29 pm
by sstrand72
Thank you! I am trying that option now - increasing it to 120 seconds to see if that remedies the problem. I am guessing that the longer the report (which ours could have hundreds of test cases) the more we should increase this to allow for a proper save of the log file? I will report back if this cures the problem we are seeing. It may take a few days, as it is not consistently happening. But if it disappears altogether, we will know we struck gold!

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Tue Jul 30, 2019 5:10 pm
by raghu.pagidi
sstrand72 wrote:
Mon Jul 22, 2019 2:29 pm
Thank you! I am trying that option now - increasing it to 120 seconds to see if that remedies the problem. I am guessing that the longer the report (which ours could have hundreds of test cases) the more we should increase this to allow for a proper save of the log file? I will report back if this cures the problem we are seeing. It may take a few days, as it is not consistently happening. But if it disappears altogether, we will know we struck gold!
Did you try with 120 seconds and got positive results on it

Re: Jenkins fails to read Junit xml for publishing Junit Test Results

Posted: Wed Jul 31, 2019 2:25 pm
by sstrand72
Here is the follow up after several overnight runs in Jenkins:

- The increase to 120 seconds did not solve the problem. But I suspect we will run into this eventually as our test reports will grow and grow.
- Here is what I changed that DID seem to solve the problem.
In Jenkins, I was renaming the Report file to something like "Report", instead of just leaving to whatever setting I had in my Ranorex job. I did this by a command override parameter in my Jenkins job. I also set the flag in the Test Suite properties to let Ranorex do the junit file and NOT have Jenkins do it.

The combination of those seemed to do the trick. Thanks for the advice!