Fail test cases when setup fails rather than skip

Ask general questions here.
Posts: 1
Joined: Wed Mar 24, 2021 9:51 pm

Fail test cases when setup fails rather than skip

Post by tmathews4 » Wed Mar 24, 2021 10:41 pm


Currently we use the Test Rail integration with our Ranorex suites. One of our suites iterates through an excel document to login to different environments and repeat the same test cases for those environments. The login functionality is currently sitting in the SETUP section of the smart folder that contains all the test cases. This is so that if the login fails it will not try to iterate through each test case, using up unwanted time finding elements we know will not be present due to the failed login. This works great, however the issue arises when reporting to test rail. Instead of reporting that the test cases failed or were skipped if one of the iterations fails its setup, Ranorex just reports to test rail that the other iterations ran fine and makes no mention of the setup failure for the iteration that failed. So while it says 100% pass in test rail, unless you go into the test plan and check iterations or open up the ranorex reports you would not notice that one iteration did not run due to setup failure.

At first I considered just moving setup into its own test case so at least that one would fail it'd at least write to test rail but then I'd either have to let the subsequent test cases run and fail if I wanted to continue using the "Continue With Iteration" error setting. If I used "Continue With Sibling" I do not believe it would run the next iterations in the excel for the other environments. The same applies to "Continue with Parent".

Is there anyway for me to either:

1) On failure in setup make it write to test rail that that iterations cases failed.
2) Create it so that if I have a test case fail with my setup logic, that it will continue to iterate through the next environments but NOT try to run the subsequent test cases for that environment since it failed to login.
3) I guess one final option would to be have a test case at end of runs that would use code to check for any failures in the run and then fail that test case so I'd at least have that marked in the Test Rail test plan, would anyone know if that is feasible and what functions to look into?

Thanks for any help.