Warnings changed to Failures from 3.0 to 3.01!? : Bug Reports

[tallahassee101]

Hi,

I have a bunch of validation tests in my recording modules that I allow to pass even if they fail. In 3.0 these were handled exactly how I would expect them to be, a warning sign showed up on the test case and the validation was highlighted in yellow and had a state of Warning.
In 3.01 these same exact tests now cause the test case (and all it's parent cases) to fail and the validation state is Failure and is highlighted in Red! Please explain to me why this change was made and where I can change it back in the settings? This ruins all of our tests and it is unacceptable!

[Support Team]

This surprises me bit. We actually consider this a bug fix because validations should either succeed or fail.

This was only a warning in V3.0.0 because "continue on fail" produces a warning in other cases (like mouse clicks). The original intent was that the validation still causes a failure, but continues execution nevertheless
(--> continue on fail). It has been like this in V2.X, and has been fixed in V3.0.1 so there is currently no option to turn it off. It is certainly possible to make this configurable, but I am not sure why you want a test case not to fail if a validation fails.

Could you describe your scenario a little better? In which cases do you validate/ignore stuff?
Do you validate in Recorder or in code? Thanks!

Regards,
Michael
Ranorex Team

[tallahassee101]

I validate in the recorder, I realize many would want a failure generated in their reports, but it would be very useful to have this configurable.

[Support Team]

but it would be very useful to have this configurable.

We already filed a feature request for that

Nevertheless, could you describe a use case where a validation makes sense that is not a real failure if it fails? We would just like to understand what the validation is good for in such a case; and if it should rather be a report action (containing some attribute value) or something similar.

Regards,
Alex
Ranorex Team

[tallahassee101]

One example is a case where a validation has found a defect in the software and a bug has been added to the database. Rather than having this test fail every time after the defect has been logged and is being fixed it would be more useful to have this part of the test continue with a warning.

Another case specific to our project is we have items called Tracks that are on our display. When you delete a Track a series of messages occur and sometimes there is a latency in this communication causing the Track to not immediately delete. We validate that the Track has been deleted and if it hasn't been deleted we want a warning and then we will wait 5 seconds before checking if the track has been deleted again, on the second time if it hasn't been deleted then we consider it a failure.

So basically I want these "warning" validation in the case of KNOWN issues and "failure" validations in all other cases.

[Support Team]

Hi,

thanks for your explanations.
As Alex mentioned before, there is already a feature request scheduled.
We're trying to find a suitable solution for this issue, but we do not have defined a time frame for implementation.

Regards,
Tobias
Support Team

[Support Team]

This feature will be available with Ranorex 3.1.0. With this version you will be able to set the report levels used when a validation fails or succeeds.

Regards,
Alex
Ranorex Team