TestSuite status incorrect on retry

[klaymore142]

Hello,

We are building a test suite in which we are using retries for each test case. In the teardown of each test case, we have a recording which checks the status of the testsuite, in order to determine if a snapshot should be captured:

Code: Select all

Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name]++;
int retryCount = ((TestCaseNode)TestSuite.CurrentTestContainer).RunRetryCount;
if (TestSuite.CurrentTestContainer.Status == Ranorex.Core.Reporting.ActivityStatus.Failed &&
    Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name] > retryCount) {
    Report.Snapshot(repo.Idea64.Self);
}

Note that Program.RUN_COUNT is a Dictionary used to determine if we are on the last retry - we don't want to create a snapshot if the test case is just going to be retried. However, there is a situation where the snapshot is saved even though the test was successful: when the test fails the first time, and then passes the second time, the Status is Failed.
Is this a bug in Ranorex? Is there a better way to only capture a snapshot on the last retry and only if the test failed on the last retry?

This is important because capturing a snapshot takes a LONG time and can actually hang the Ranorex process (we suspect memory issues).

[odklizec]

Hi,

To get the TestSuite status, you need to use this code:

Code: Select all

var tcStatus = Ranorex.Core.Reporting.ActivityStack.Current.Status; // returns test suite status

However, as far as I remember, this code does not work in TestSuite Teardown section? Please check this post:
https://www.ranorex.com/forum/testsuite ... tml#p55894
What I think you can do, is to get the status of particular test case, using code like this:

Code: Select all

var tcStatus = TestSuite.Current.GetTestContainer("tcName").Status; // get status of specified TC 

This should work even in TestSuite teardown section.

[klaymore142]

Unfortunately, this doesn't work. GetTestContainer(string).Status has the same behavior as CurrentTestContainer.Status - when the test fails on the first try, but succeeds on retry, both statuses are Failed.

[odklizec]

Hi,

At first, what Ranorex version do you use? Please make sure, you are using most recent 9.2.1! Debugging your problem with anything older is just a waste of time and energy

At next, are you sure the code I suggested is located at appropriate place?

This code must be placed in Program.cs (right after error = TestSuiteRunner.Run line)...

Code: Select all

var tcStatus = Ranorex.Core.Reporting.ActivityStack.Current.Status; // returns test suite status

And this code must be placed somewhere after the testcase/smartfolder with retry count (e.g. in Test Suite TearDown section).

Code: Select all

var tcStatus = TestSuite.Current.GetTestContainer("tcName").Status; // get status of specified TC 

[klaymore142]

We are on 9.2.0, haven't updated to 9.2.1 yet. Is there a change in 9.2.1 which would affect this?

Adding these lines to Program.cs after the call to TestSuiteRunner.Run() doesn't help, since I need to get the status in the teardown of the failed test case. I need a snapshot for failed test cases, at the moment that the test failed, so that I can debug why tests failed when they do. There's multiple tests, and we need to continue running additional tests if some fail.

[RobinHood42]

Hi!

You can check the status of the test case without using a teardown section. I mean put the recording that checks the test case status at the very end of the test case. If the test case succeeds after retrying, the status should be "success".

var tcStatus = TestSuite.Current.GetTestContainer("TestCase").Status;

Cheer,
Robin

[klaymore142]

Hi Robin,

Although we can check the test status without a teardown section, a recording added to the test case after a failing recording does not get executed. For instance, if our test suite looks like this:

• SuiteName
  • Setup
  • Recording1
  • Snapshot
  • Teardown

If Recording1 fails, "Snapshot" does not execute, regardless of the setting for "Error Behavior". It only executes if "Snapshot" is added to the Teardown. This is likely because all test failures are handled as Exceptions, which immediately ends the test and jumps straight to the Teardown.

[klaymore142]

We found a solution that works with retries, and does indeed indicate the "latest" status of the current container: Ranorex.Core.Reporting.TestReport.CurrentTestContainerActivity.Status

So, in the teardown of each test, the first recording module calls the following function:

        public static void TakeSnapshotIfFailed()
        {
            int retryCount = ((TestCaseNode)TestSuite.CurrentTestContainer).RunRetryCount;
            if (!Program.RUN_COUNT.ContainsKey(TestSuite.CurrentTestContainer.Name)) {
                Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name] = 0;
            }
            Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name]++;
            if (Ranorex.Core.Reporting.TestReport.CurrentTestContainerActivity.Status == Ranorex.Core.Reporting.ActivityStatus.Failed &&
                Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name] > retryCount) {
                Report.Log(ReportLevel.Info, "User", "Test failed - capturing snapshot");
                Report.Snapshot(repo.Idea64.Self);
            }
        }

[odklizec]

Hi,

Nice to hear you found the way. Thanks for sharing the solution!