Bug in JUnit reports

Bug reports.
matthewk
Posts: 15
Joined: Wed Aug 23, 2017 12:10 pm

Bug in JUnit reports

Post by matthewk » Wed Aug 23, 2017 12:13 pm

The JUnit reports produced don't seem to be marking tests as failures.

Running in TeamCity failed test cases are being marked as OK.

Looking at the output xml it does not have a failure xml element produced for failed test cases

McTurtle
Posts: 297
Joined: Thu Feb 23, 2017 10:37 am
Location: Benedikt, Slovenia

Re: Bug in JUnit reports

Post by McTurtle » Fri Aug 25, 2017 2:16 pm

Hello matthewk,

Could you post one such example JUnit report and the corresponding *.rxlog file created during the same test execution? Also, which version of Ranorex are you using and how are you making the transformation? The old JUnit transformation file no longer works with newer versions of Ranorex. There is now an option for this in Ranorex Studio: JUnit-compatible reports

Does it work if you execute the test locally without TeamCity or is the JUnit report also incorrect?

Regards,
McTurtle

matthewk
Posts: 15
Joined: Wed Aug 23, 2017 12:10 pm

Re: Bug in JUnit reports

Post by matthewk » Wed Sep 06, 2017 11:25 am

I don't have files available to share right now. This was using the built in JUnit report option in Ranorex 7.1 and not any custom transforms.

What happens is when a test case fails it adds an <error></error> element in the JUnit xml file.

Failed test cases should use a <failure></failure> element instead as there are differences between errors & failures for a lot of CI tools (TeamCity in our case).

The outputted JUnit xml file is the same whether executed locally or through the CI.

I have had emails back and forth with Ranorex and they are looking in to it - but in the meantime I have a workaround where I parse the generated file and replace the values myself - which is working well for our reporting needs

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Bug in JUnit reports

Post by Support Team » Thu Sep 07, 2017 12:04 pm

Hello matthewk,

As you have already found out, errors and failures are not treated in the same way.
I would suggest that you get in contact with us again via the previous email conversation so that we can continue working on this issue.

If you are unable to find the previous email conversation, you can also open a new ticket by sending an email to [email protected].
Please don't forget to attach your currently valid customer ID to the email.

We are looking forward to you email.

Sincerely,
Tomaž

tfabris
Posts: 2
Joined: Tue Sep 26, 2017 7:04 pm

Re: Bug in JUnit reports

Post by tfabris » Tue Sep 26, 2017 7:16 pm

matthewk, this might help.

We have discovered the root cause of two major Junit XML reporting bugs related to Ranorex integrating with Team City. Both bugs are in the gray area realm where it is hard to decide if the bug is Ranorex’s bug or Team City’s bug. Is the bug in the writing of the Junit file, or in the parsing of the Junit file? I don’t know.

I have reported these details to Ranorex Tech Support and Team City tech support both. Let's see if they can work together to solve them? :D

I have details of both bugs below, including functional workarounds.



Bug: Team City reports Ranorex test cases as “passed” in the Team City test results, even though the test cases actually failed.

Cause:
- Ranorex has a feature for auto-saving reports to disk every 30 seconds. The feature is located in the Report tab of the Test Suite Properties, and is turned on by default.
- Team city has a feature for automatically reading and displaying Junit reports by parsing the report files on disk.
- Both processes work at the same time. Team city attempts to read the report file and interpret each test case as it’s being written out by Ranorex.
- The processes working at the same time is an intended feature, so that Team City can report live real-time individual test case results before the entire test suite is complete. This is an important feature of Team City: You can watch the tests run and get early notification of test case failures in time-critical situations.
- Speculation: Ranorex somehow writes the file in such a way that, every 30 seconds, it’s writing data about the test case but without the failure information (since the test case hasn’t failed yet). Perhaps it writes preliminary information about the start of the test case into the file, but then later, after the test case fails, it updates the XML to indicate that the test case failed. Team city picks up only the preliminary information, using the “first” information it was given, and assumes that the test case passed.

Repro:
- Create a test suite in Ranorex with multiple test cases inside it, and where each of the test cases takes 60 seconds or more to run.
- Configure the reporting for the test suite in Ranorex to automatically save the reports every 30 seconds.
- Create a Team City build which launches the Ranorex test suite.
- Add the /ju parameter to Team City command line which launches the test suite so that the Junit files are generated along with the ranorex report files. Make sure the report files are going to a known location, such as the Team City checkout directory.
- Configure the Team City build, in the “Build Features” screen, for Junit XML report processing, so that it searches for .junit.xml files in the known location.
- Run the team city build.
- Deliberately fail one of the test cases by interfering with it during the test run.
- After it is done, look at the build results in Team City. Open the “TESTS” tab in the build results.
- Team city “TESTS” tab will intermittently report that all test cases passed (every test case will say “OK”) even though at least some of them failed.

Workaround:
- In the Ranorex test suite, disable the report auto-save feature by entering zero seconds instead of 30 seconds for auto save, located in the Report tab of the Test Suite properties.
- This workaround is problematic because it means that Team City feature no longer works as intended. It can no longer can report live real-time pass/fail results for each test case as it happens.





Bug: Team city gets error “Premature end of file” when parsing Junit XML report files written out from a Ranorex test suite.

Cause:
- This occurs when a Ranorex test suite ran no test cases, and thus writes no individual test case results to the Junit XML file.

Workaround:
- Make sure all Ranorex test suites contain at least one test case, and report at least one test result.
- Workaround is problematic because perhaps it was intentional. For example, perhaps that particular test run had deliberately skipped or ignored all tests, perhaps due to conditional code in the test cases sensing that the test was not applicable at the time.

User avatar
Support Team
Site Admin
Site Admin
Posts: 12145
Joined: Fri Jul 07, 2006 4:30 pm
Location: Houston, Texas, USA
Contact:

Re: Bug in JUnit reports

Post by Support Team » Tue Oct 03, 2017 1:11 pm

Hello tfabris,

We would like to analyze these issues more in dept. Could you please send us an email to [email protected] so that we could get in contact directly?
Please do include your customer ID in your email and mention this forum post.

We are looking forward to your email.

Sincerely,
Tomaž

tfabris
Posts: 2
Joined: Tue Sep 26, 2017 7:04 pm

Re: Bug in JUnit reports

Post by tfabris » Tue Oct 03, 2017 6:31 pm

Tomaž,

Thanks very much for your reply!

We already have engaged with the Ranorex support on both issues. The existing support tickets for these two separate issues are:

The "Premature end of file" Junit XML parsing error is reported by Team City when a Ranorex suite has no test results in it.
This one is reportedly fixed in Ranorex 7.2. We haven't upgraded to that version yet, and we are successfully working around the issue for now.

Team City Junit XML parsing reports Ranorex tests as "passed" even when the tests failed.
This one is not yet fixed according to the email conversation we have had with support. We are successfully working around the issue by disabling the "Auto save reports every 30 seconds" feature of Ranorex.