Possible to notify at point of failure? : General Questions

[Gav]

Hi,

We're evaluating Ranorex and it's looking pretty good. We're now at the stage of determining how we can handle recovery scenarios and notifications.

We are running a suite of tests, with each recording configured to try to continue its steps if there's a failure. Each case in the suite is then configured to continue to the next case if the recordings can't continue.

But rather than wait until the suite of tests has finished to get a notification and report (e.g. by email using the Send Mail module) is it possible to have an alert as soon as there's a failure in any of the recordings or cases?

Thanks,

Gav

[Support Team]

Hi,

But rather than wait until the suite of tests has finished to get a notification and report (e.g. by email using the Send Mail module) is it possible to have an alert as soon as there's a failure in any of the recordings or cases?

If I can follow your description you want a notification if something fails, right? Or do you want a message box which pauses the automation?

Regards,
Peter
Ranorex Team

[Gav]

Hi Peter,

That's right - we would like the tests to continue without pause, but to 'raise a flag' as soon as something fails so we can start working on it, rather than waiting for the report at the end.

Gav

[sdaly]

I like this idea too

If the ranorex assertions fire an even upon failure, we could add a handler for that event and ping out an email...

[Support Team]

Hi,

That's right - we would like the tests to continue without pause, but to 'raise a flag' as soon as something fails so we can start working on it, rather than waiting for the report at the end.

Currently it is possible to use the Ranorex Blog Customizing Ranorex Reports or you use the Tear Down part of the Test Case to handle this.

If the ranorex assertions fire an even upon failure, we could add a handler for that event and ping out an email...

It would be possible to implement an EventHandler, but currently we don't know what this handler should do? Should this handler allow to execute "whatever" code or just logging code? Only on failure or even on success?

Would it be possible to post us a few samples, where the other two solutions aren't enough and you need an EventHandler?

Thank you.

Regards,
Peter
Ranorex Team

[sdaly]

Hi Peter

The way I would imagine to use it would be like this -

Code: Select all

//register a handler for the validate 'failure' event
Validate.Failure += new EventHandler(Failure);

public static void Failure(object sender, RanorexEventArgs e){
sendEmail("Assertion failed - location is " + e.className + "::"  + e.memberName + " - " + e.errorMessage);   
}

[sdaly]

...I should probably explain this a bit more!...

We use NUnit to control our tests - we determine failure by surrounding each test in a try/catch block. I would like to send out an email notification upon failure. We could add some code to our tear downs to send an email if the test has failed. We would have to remember to add this bit of logic to each of our teardowns though.

The advantages I see of having an event fired upon failure is -
>Say we have a validate.exists within a code module/helper method such as "Login()"...if a 'failure' event was fired here (with stacktrace details) which invokes a method (our handler) that sends an email, then we would get an email to say failure has occurred at 'Login' and not just that test x has failed.
>We could log failures to our DB for each validate, again with the class/member details...this would allow us to report on failing areas such as 'Login', instead of just being able to see failures at test level. We could quickly see 10 tests have failed but 10 'Login's have failed and therefore identify that there is a login issue.
>If we have a test which has several validates wrapped in a try/catch because we want it to try all steps, then we would get an email notification for each of the failure points, not just one big failure at the end of the test.

In general, it would just be a nice little feature which would open up some powerful opportunities...

I hope that makes sense

Sorry for hijacking this thread...all seems relevant though

[Support Team]

Hey Scott,

We discussed this internally and we could implement an event handler.
But actually we don't know which arguments we should return.
We could return the first class/method which forces the validate? We could even return you the stacktrace, but for the most users the stacktrace is too complex and difficult to handle and thus not useful.

So what would you like to know when such an event handler is called?

Regards,
Peter
Ranorex Team

[sdaly]

Hi Peter

I guess the class/method which triggered the validate would be adequate and would keep it simple...so that, the validation message and the exception message?

Cheers
Scott

[Support Team]

I added a feature request to our internal bug tracking system.

the validation message and the exception message?

You want the validation exception and the EventHandler exception inside the Report, right?

Regards,
Peter
Ranorex Team

[sdaly]

Hi Peter

Those are the things I would like passed through the event args! So then in the custom event handler, you can do whatever!

Thanks
Scott

[Gav]

Please excuse my noobness, but if the feature request goes ahead, does it mean our developer guys will be able to get Ranorex to send notifications as soon as there is a failure in the test suite, while the test suite then attempts to carry on with other tests?

Gav

[Support Team]

This feature request would just provide another way to react to validation failures using code, e.g. by sending an email.

does it mean our developer guys will be able to get Ranorex to send notifications as soon as there is a failure in the test suite, while the test suite then attempts to carry on with other tests?

That feature is already available. You just need to attach a custom logger that sends an email if a failure is reported. See following blog on how to customize Ranorex reports and the "Send Mail" automation module:
http://www.ranorex.com/blog/customizing-ranorex-reports
send-mail-module-t2356.html

Regards,
Alex
Ranorex Team

[Gav]

Hi Alex,

Apologies again, but I just need to be absolutely clear on this before I get our devs to have a look at it for me. Will the custom logger allow a message to be sent immediately that a step fails (regardless that the script will attempt to continue), without having to wait until the suite completes and the report containing failures and results is presented at the end?

Gav

[sdaly]

Gav, that is correct, you can do that. I implemented a solution using a custom logger last week (never even knew it existed!)... I'm going to write a blog post next week on it so I'll post you a link once I've done so...