Incorrect Error Count

Bug reports.
tallahassee101
Posts: 169
Joined: Thu Jan 13, 2011 2:06 pm

Incorrect Error Count

Post by tallahassee101 » Thu Nov 29, 2012 4:59 pm

Hi,

I've attached 3 images for a test report that I generated with Ranorex. As you'll see in the Error Report file the "failed" test count is 6; however if you look at the other two images you'll see that only 2 recordings failed. We are trying to use the failed count as a metric for the current stability of our product, however this is not possible with the current incorrect failure tally method.
So I have two questions. One, could you explain how this failure count is reached specifically what constitutes a failure in this report. Two, is there any way we can change how this failure tally is generated? We'd like it to only count a recording as a failure ( test cases and iterations should not count as a failure, since the recording underneath them already count ). We are using Ranorex 3.3.3.

Thank you
-Nick
Attachments
symbologyerrors3.png
Error Report
symbologyerrors3.png (69.52 KiB) Viewed 1132 times
symbologyerrors2.png
Error 2
symbologyerrors2.png (70.76 KiB) Viewed 1132 times
symbologyerrors.png
Error 1
symbologyerrors.png (67.55 KiB) Viewed 1132 times

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Incorrect Error Count

Post by Support Team » Fri Nov 30, 2012 11:20 am

Hello,

If a test case fails the failure count will be incremented.
There could be many reasons for failing a test: e.g. a validation fails, elements are not found etc.

Unfortunately, there is no way to change this failure count since it is implemented in Ranorex.Core.
We will discuss this feature request with our developers.

You could customize your report.
Please take a look at section Customizing Ranorex Reports in our blog.

Regards,
Markus (T)
.
Image

tallahassee101
Posts: 169
Joined: Thu Jan 13, 2011 2:06 pm

Re: Incorrect Error Count

Post by tallahassee101 » Mon Dec 03, 2012 2:40 pm

Markus,

How would customizing my report help solve the issue with the failure count? Is it clear how the error count is being improperly reported in the screenshots I attached? I understand there may not be a way to change the failure count, however I am reporting this as a bug in the ranorex software not a feature request for modifying failure counts. I'd like to see the correct error count on my ranorex reports, which in my opinion should be 2 (not 6) for the TacViewSymbology test suite report I attached.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Incorrect Error Count

Post by Support Team » Mon Dec 03, 2012 3:12 pm

Hello,

You could customize your reports in order to get such a failure count but it would result in a big effort.

Each Test Case is designed as one unit which can fail or pass.
If one recording failed within a Test Case the whole Test Case would fail.

In your case, you have parent and child Test Cases. If one child Test Case failed the parent Test Case would fail as well. Both failed Test Cases (parent and child) would increment the failure count.

Regards,
Markus (T)
.
Image

tallahassee101
Posts: 169
Joined: Thu Jan 13, 2011 2:06 pm

Re: Incorrect Error Count

Post by tallahassee101 » Mon Dec 03, 2012 3:30 pm

Markus,

Thank you for the explanation. I'd like to report the parent/child failure as a bug. I can't think of a scenario where we would ever want to count that as two separate failures. It makes the parent/child feature of test cases not work for organizing tests appropriately.

Thanks,
-Nick

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Incorrect Error Count

Post by Support Team » Mon Dec 03, 2012 3:40 pm

Hello,

Thank you for your information.
We will discuss this internally.

Regards,
Markus (T)
.
Image