TestSuite status incorrect on retry

Ask general questions here.
klaymore142
Posts: 6
Joined: Wed Jul 18, 2018 6:56 pm

TestSuite status incorrect on retry

Post by klaymore142 » Sat Dec 14, 2019 1:09 am

Hello,

We are building a test suite in which we are using retries for each test case. In the teardown of each test case, we have a recording which checks the status of the testsuite, in order to determine if a snapshot should be captured:

Code: Select all

Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name]++;
int retryCount = ((TestCaseNode)TestSuite.CurrentTestContainer).RunRetryCount;
if (TestSuite.CurrentTestContainer.Status == Ranorex.Core.Reporting.ActivityStatus.Failed &&
    Program.RUN_COUNT[TestSuite.CurrentTestContainer.Name] > retryCount) {
    Report.Snapshot(repo.Idea64.Self);
}
Note that Program.RUN_COUNT is a Dictionary used to determine if we are on the last retry - we don't want to create a snapshot if the test case is just going to be retried. However, there is a situation where the snapshot is saved even though the test was successful: when the test fails the first time, and then passes the second time, the Status is Failed.
Is this a bug in Ranorex? Is there a better way to only capture a snapshot on the last retry and only if the test failed on the last retry?

This is important because capturing a snapshot takes a LONG time and can actually hang the Ranorex process (we suspect memory issues).

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 5459
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: TestSuite status incorrect on retry

Post by odklizec » Mon Dec 16, 2019 1:34 pm

Hi,

To get the TestSuite status, you need to use this code:

Code: Select all

var tcStatus = Ranorex.Core.Reporting.ActivityStack.Current.Status; // returns test suite status
However, as far as I remember, this code does not work in TestSuite Teardown section? Please check this post:
https://www.ranorex.com/forum/testsuite ... tml#p55894
What I think you can do, is to get the status of particular test case, using code like this:

Code: Select all

var tcStatus = TestSuite.Current.GetTestContainer("tcName").Status; // get status of specified TC 
This should work even in TestSuite teardown section.
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

klaymore142
Posts: 6
Joined: Wed Jul 18, 2018 6:56 pm

Re: TestSuite status incorrect on retry

Post by klaymore142 » Tue Dec 17, 2019 11:02 pm

Unfortunately, this doesn't work. GetTestContainer(string).Status has the same behavior as CurrentTestContainer.Status - when the test fails on the first try, but succeeds on retry, both statuses are Failed.

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 5459
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: TestSuite status incorrect on retry

Post by odklizec » Wed Dec 18, 2019 9:18 am

Hi,

At first, what Ranorex version do you use? Please make sure, you are using most recent 9.2.1! Debugging your problem with anything older is just a waste of time and energy ;)

At next, are you sure the code I suggested is located at appropriate place?

This code must be placed in Program.cs (right after error = TestSuiteRunner.Run line)...

Code: Select all

var tcStatus = Ranorex.Core.Reporting.ActivityStack.Current.Status; // returns test suite status
And this code must be placed somewhere after the testcase/smartfolder with retry count (e.g. in Test Suite TearDown section).

Code: Select all

var tcStatus = TestSuite.Current.GetTestContainer("tcName").Status; // get status of specified TC 
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

klaymore142
Posts: 6
Joined: Wed Jul 18, 2018 6:56 pm

Re: TestSuite status incorrect on retry

Post by klaymore142 » Sat Dec 21, 2019 12:24 am

We are on 9.2.0, haven't updated to 9.2.1 yet. Is there a change in 9.2.1 which would affect this?

Adding these lines to Program.cs after the call to TestSuiteRunner.Run() doesn't help, since I need to get the status in the teardown of the failed test case. I need a snapshot for failed test cases, at the moment that the test failed, so that I can debug why tests failed when they do. There's multiple tests, and we need to continue running additional tests if some fail.

User avatar
RobinHood42
Posts: 311
Joined: Fri Jan 09, 2015 3:24 pm

Re: TestSuite status incorrect on retry

Post by RobinHood42 » Mon Dec 23, 2019 1:07 pm

Hi!

You can check the status of the test case without using a teardown section. I mean put the recording that checks the test case status at the very end of the test case. If the test case succeeds after retrying, the status should be "success".
var tcStatus = TestSuite.Current.GetTestContainer("TestCase").Status;
Cheer,
Robin

klaymore142
Posts: 6
Joined: Wed Jul 18, 2018 6:56 pm

Re: TestSuite status incorrect on retry

Post by klaymore142 » Fri Jan 03, 2020 12:12 am

Hi Robin,

Although we can check the test status without a teardown section, a recording added to the test case after a failing recording does not get executed. For instance, if our test suite looks like this:
  • SuiteName
    • Setup
    • Recording1
    • Snapshot
    • Teardown
If Recording1 fails, "Snapshot" does not execute, regardless of the setting for "Error Behavior". It only executes if "Snapshot" is added to the Teardown. This is likely because all test failures are handled as Exceptions, which immediately ends the test and jumps straight to the Teardown.