Exception reporting in Ranorex Report

Bug reports.
Saha
Posts: 39
Joined: Tue Mar 01, 2011 6:20 pm

Exception reporting in Ranorex Report

Post by Saha » Wed Aug 24, 2011 4:51 pm

Hi,

Whenever I have exception from Ranorex in my script, script shows up in green in Reports. Unless I open the script I would not know that my script is failing.
Am I missing some configuration here? How to show scripts failing in red when there is any exception?

Thanks,
Saha
Attachments
RanorexReport.JPG
Screenshot
RanorexReport.JPG (60.9 KiB) Viewed 1788 times

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Exception reporting in Ranorex Report

Post by Support Team » Thu Aug 25, 2011 9:36 am

Hi,
Saha wrote:Am I missing some configuration here? How to show scripts failing in red when there is any exception?
If you use Report.Error() inside your user code module then this is the expected behavior, because Report.Error() doesn't affect the Report. You have to log Report.Failure() if you want to set the test case to the failstate too.

Regards,
Peter
Ranorex Team
.
Image

Saha
Posts: 39
Joined: Tue Mar 01, 2011 6:20 pm

Re: Exception reporting in Ranorex Report

Post by Saha » Thu Aug 25, 2011 4:58 pm

This exception in screenshot is not coming from Report.Failure or Report.Error.
It is happening when Ranorex is not able to find element in UI because the expected page was not loaded.
If you see the screenshot it is an ElementNtofoundException

Thanks,
Saha

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Exception reporting in Ranorex Report

Post by Support Team » Fri Aug 26, 2011 11:40 am

Saha wrote:This exception in screenshot is not coming from Report.Failure or Report.Error.
It is happening when Ranorex is not able to find element in UI because the expected page was not loaded.
If you see the screenshot it is an ElementNtofoundException
If you get an ElemenNotFound Excpetion you should get a failure and not an error. Would it be possible to post us the expected user code file to check the code? Are you using a Recorder Module or a User Code Module?

Regards,
Peter
Ranorex Team
.
Image

anja
Posts: 67
Joined: Fri Nov 26, 2010 3:39 pm

Re: Exception reporting in Ranorex Report

Post by anja » Fri Jul 20, 2012 7:38 am

Hi Ranorex-Team,

we've got a similar problem with our Report... at some point we have a "report.Error" message.
The report stays green at this Testcase and iteration. You said in your previous post, that error has no effect on the report -> so this behaviour is as you would expect....
But in another testcase I saw that "error" has an influence. Within the attached image you see this part of the report:
Ranorexreport1.JPG
Ranorexreport1.JPG (96.46 KiB) Viewed 1594 times
Within this testcase, no other module or iteration is performed.
The step and the iteration are marked red, even if the error message is an error and not a failure.
The only difference is the Category... once it is written by the user and another time by the module... why should there be a difference in the behaviour? Both are errors...

For us, it would be nice to see, if a module got an error message on the testcase level, because if an error occurs, you should be aware that this error is there. Maybe it could be marked different as the failure but it should be visible from the upmost level.

Best regards
Anja

User avatar
artur_gadomski
Posts: 207
Joined: Mon Jul 19, 2010 6:55 am
Location: Copenhagen, Denmark
Contact:

Re: Exception reporting in Ranorex Report

Post by artur_gadomski » Fri Jul 20, 2012 8:44 am

Me too would like to see Errors change test cases they belong too. Making them look like failed cases or some other 'highly visible' (as opposed to tiny yellow on white icon for warnings) indication that something in test case went wrong. We would like to be able to say that test case fails if one of our designed validations fails ie. expected 6 got 5. We would like to use Errors when typical flow of test case was interrupted, ie. login form that we expect to be there wasn't.
After a period of stabilization where almost each test run fails and a period of stable run where each test run pass people(me) tend to skim over passed (green) test cases without even opening them trusting that if it's green it wen't fine. But there could be errors there. For now we resorted to using failure everywhere, and editing xsl style sheet so that errors are more visible is not really an option for us. We don't have an xsl expert and default xsl is not the simplest one to get into.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Exception reporting in Ranorex Report

Post by Support Team » Fri Jul 20, 2012 10:11 am

Hi,
Maybe it could be marked different as the failure but it should be visible from the upmost level.
I agree that it would be nice to see errors at first glance in the report files (in a way it already was implemented for "Warnings").

I also agree that it might be confusing to see errors in the report which sometimes result in a failed testcase and sometimes in a successful one.
Maybe this explanation will clear up some things.

A testcase fails if an exception is thrown, there is a module error or you throw a report.failure message. Module errors are thrown from the system and occur e.g. if one or more actions have failed or an exception was thrown. So module errors result in a failed testcase. (Here the term "Error" in the reportlevel may be an inartfully expression.)

To sum up, testcases only fail if there is a module error or a failure. All other errors will result in a success.
We would like to be able to say that test case fails if one of our designed validations fails ie. expected 6 got 5. We would like to use Errors when typical flow of test case was interrupted, ie. login form that we expect to be there wasn't.
This shouldn't be a problem since you are able to set/change the "Report Level On Failure / Success" in the validation-properties.

Kind regards,
Larissa
Ranorex Support Team
.
Image

anja
Posts: 67
Joined: Fri Nov 26, 2010 3:39 pm

Re: Exception reporting in Ranorex Report

Post by anja » Fri Jul 20, 2012 12:16 pm

Hi,

thank you for your very fast answer.

I'm not sure if I understood your answer completely (or the message behind):
You agree, that it would be better to see it if there is an "Error". But is there something happening? Will you implement something in this directionor even discuss internally a change in the behaviour?

It would be really great, because we have many Report.Error commands in our Code and just now recognized that an Error is not visible in a successfull report... It would be much effort to change our code to only report Failures... (Many different Solutions for different projects will be affected...)

In my opinion, it makes no sence to mark tests which results in an error as passed... if an error occurs or a failure, both should result in a failed test...
If I expect an error to occure (e.g. a validation to fail) I'm responsible to catch the resulting exception and handle it as successfull.

As Arthur had written, we have defined in which situations an error or a failure should be reported. Now we have to decide if we need to change this back to reporting everywhere failures and then we need another possibility to make it visible in what situation the failure occured....

We need your answer for deciding how we can go on...

Thank you.

Best regards
Anja

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Exception reporting in Ranorex Report

Post by Support Team » Mon Jul 23, 2012 9:40 am

Hi Anja,
Will you implement something in this directionor even discuss internally a change in the behaviour?
I already talked to the development-team. They will do their best to make it more user-friendly.
In my opinion, it makes no sence to mark tests which results in an error as passed... if an error occurs or a failure, both should result in a failed test...
Maybe I should explain what it means if a testcase ends with success:
A successful end of a testcase doesn't imply a 100 % correct run of your application under test. It just says that it was possible to run the whole testcase.

So an error-message just tells that something in your construction of your test was not well considered. Nevertheless the testcase can be a success.

Only if it is not possible to run the whole testcase (because the error appeared in the application was too fatal, e.g. an exception was thrown), the testcase fails and ends with a red light.

I hope this explanation makes it more obvious for you why a green light makes sense although there might be errors.

Kind regards,
Larissa
Ranorex Support Team
.
Image