What strategy do you use for disabling tests that are failing for reasons outwith your control?

Best practices, code snippets for common functionality, examples, and guidelines.
User avatar
Stub
Posts: 268
Joined: Fri Jul 15, 2016 1:35 pm

What strategy do you use for disabling tests that are failing for reasons outwith your control?

Post by Stub » Fri Jun 21, 2019 3:10 pm

One of my tests suites has a fair few test cases in it that all pass with Ranorex v8.2.1.
RanorexProgress.png
RanorexProgress.png (2.44 KiB) Viewed 169 times
But as of Ranorex v8.3.3 there are a few of these that I need to DISable because they're failing due to changes in Ranorex itself.

Normally I use Run Configurations so that I can turn tests on/off as required. But with my day to day Ranorex development configuration I am always turning test folders on/off depending on whatever I'm working on.

How do you folks deal with test cases that you do not want to run (because they are guaranteed to fail and there is nothing you can do about that at the moment), but do not want to delete (because you hope that the underlying issue will be resolved)?

We've got another case where a feature in our AUT changed, so a set of tests for that feature need to be rewritten or significantly adjusted to suit. One whole year on and I'm still waiting on that duder to fix it up! I have our main test run configuration set to ignore that area, but again, I have to keep DISabling the tests in this area when I'm working on other things.

So I'm wondering if I'm approaching this incorrectly. I've just got so fed up of this problem - because I've been doing some Ranorex software upgrading recently I've been butting up against this annoyance repeatedly - that I've bunged in a code module to each of the failing test cases to skip them:

Code: Select all

TestSuite.CurrentTestContainer.Checked = false;

User avatar
Franz
Posts: 3
Joined: Fri Jun 21, 2019 3:25 pm

Re: What strategy do you use for disabling tests that are failing for reasons outwith your control?

Post by Franz » Mon Jun 24, 2019 3:27 pm

If you have a [Setup] in every TestCase you could add a CodeModule that checks the name of the current running TestCase, then just validate if the name appears in your string[] affectedTestCases = { "TestCase1", "TestCase2", "TestCase3", }; array and then skip by code if necessary.

A better option would be to write a plugin that overrides the *.rxtst file after (re)building your solution. In this case i would work with C# XDocument and delete the <TestContainer> entries.