Setting up Test Case dependencies

Ranorex Studio, Spy, Recorder, and Driver.
uhmdown
Posts: 54
Joined: Mon Apr 03, 2017 12:00 pm

Setting up Test Case dependencies

Post by uhmdown » Thu Apr 27, 2017 12:17 am

I have some cases where I only want to execute modules in a testcase/folder if some previous test module didn't fail. Basically, some tests only make sense if some previous test didn't fail.

So, something like this:
if(no error in testmodule_A)
{ run testmodule_B}
else
{skip testmodule_B, possibly with a warning}

I'm sure this can be done by tracking execution progress manually in the test modules in question, but I might be reinventing the wheel.

I can't find support for this in Ranorex Studio out-of-box. There is some very crude error-behaviour configuration for testcases/smartfolders. Are there any more flexible ways to set up dependency chains for test modules?

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 7470
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Setting up Test Case dependencies

Post by odklizec » Thu Apr 27, 2017 8:35 am

Hi,

At the moment, there is no out-of-the-box solution for this. However, you can add such logic via code. An example how to enable/disable test cases via code, can be found for example here:
skip-module-during-runtime-t10822.html#p43402

Basically, you will have to add a code module between each TestCase (or in TearDown section of each TC) and in this code module you will have to evaluate the test case state (success/failed) and based of this, enable or disable next TC. Hope this helps?

BTW, it seems that the conditional execution of test modules is something Ranorex folks are already working on...
https://www.ranorex.com/company/product-roadmap.html
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

uhmdown
Posts: 54
Joined: Mon Apr 03, 2017 12:00 pm

Re: Setting up Test Case dependencies

Post by uhmdown » Tue May 02, 2017 8:20 am

I figured it would be something like that.

I was considering building a map over dependencies in a CSV file, load it in at the beginning of the test, and then have testmodules check whether its dependencies are fullfilled.

But considering that Ranorex will come out with an out-of-box solution soon, I'll wait and see.

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Setting up Test Case dependencies

Post by krstcs » Tue May 02, 2017 1:37 pm

Ranorex already does this automatically, except for the warning. If you put your modules in the same test case and module A fails, Ranorex will skip the rest of that test case iteration (so the other modules after the failure won't execute for that iteration). And, using Error Behavior settings, you can make Ranorex do different things with the rest of the iterations or the rest of the test.

Was the warning specifically what you needed?
Shortcuts usually aren't...

uhmdown
Posts: 54
Joined: Mon Apr 03, 2017 12:00 pm

Re: Setting up Test Case dependencies

Post by uhmdown » Tue May 02, 2017 2:03 pm

If you put your modules in the same test case
That's where the problem is.
Basically, I would like there to be an Error Behaviour option where you can define an arbitrary set of test cases (maybe also test modules) anywhere in the TestSuite to be skipped if the current test fails.
The current Error Behavior options are simply too inflexible.

I have some Sanity Checks test cases performed at the very beginning of the Test Suite, and depending on which ones fail, some other test cases somewhere else in the (not necessarily next to, or in the same folder) should then be skipped.

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Setting up Test Case dependencies

Post by krstcs » Tue May 02, 2017 3:46 pm

You can achieve that by nesting your test cases (6.X and earlier)/smart folders (in 7.0).

[TestSuite]
--[TestCase1]
----SanityModules
----[SmartFolder1]
-------TestsToSkipIfSanityFails

If Sanity fails, SmartFolder1 will not run.
Shortcuts usually aren't...

uhmdown
Posts: 54
Joined: Mon Apr 03, 2017 12:00 pm

Re: Setting up Test Case dependencies

Post by uhmdown » Tue May 02, 2017 4:25 pm

It's still all-or-nothing.
You're right that in a small handful of cases, you can set up some basic dependency checks.
But consider this:

[TestSuite]
--[TestCase1]
----SanityModules
----[SmartFolder1]
-------Test should still run
----[SmartFolder2]
-------TestsToSkipIfSanityFails
----[SmartFolder3]
-------Test should still run

The Error Behaviour that achieves what you described would skip all the smartfolders, not just the ones that depended on the specific sanity modules that failed.

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Setting up Test Case dependencies

Post by krstcs » Tue May 02, 2017 4:47 pm

More nesting will again help with this, as well as reworking where each folder is in the test case.

If you're putting things in different folders then that usually means they aren't directly related (as in SF 1 isn't necessary for SF2). If they are directly related, then you may need to restructure your test to make it simpler and more clear, potentially using Setup and Teardown in each TC/SF to make sure that the environment is setup correctly for each unit.

[TestSuite]
--[TestCase1]
----[SmartFolder4]
------SanityModules
--------[SmartFolder2]
----------TestsToSkipIfSanityFails
----[SmartFolder1]
------Test should still run
----[SmartFolder3]
------Test should still run

Here, Sanity would still be run before SF1 & SF3, but SF2 would need to be in the same folder (SF4).

Remember, you can also split out sections into groups and then re-use them in multiple locations easily.

In addition, you can split out each business unit logic into it's own project under the same solution so that each one will be specific for the use case being tests.

There are several more ways that this can be done, if you think outside the box a bit and perhaps rethink the test methodology.

But, if this is a case where you feel Ranorex does not provide functionality you think it should have, you might want to look at their Uservoice page (uservoice.ranorex.com). Make sure to search first to make sure there isn't anything related to what you are looking for, and then post a suggestion if not. I do know that they have conditional module execution on the list of features to implement, so perhaps that will help you out.
Shortcuts usually aren't...

uhmdown
Posts: 54
Joined: Mon Apr 03, 2017 12:00 pm

Re: Setting up Test Case dependencies

Post by uhmdown » Tue May 02, 2017 6:52 pm

As odklizec pointed out, some kind of dependency management appears to already be in the works.

I totally get what you're saying, and I thought about that possibility too after you mentioned nesting the folders; I'm sure I could creatively restructure my Test Suite so I can get some kind of dependency chain, but I find that approach is a bit backwards here.
I've structured the Test Suite in a way so it gives me the overview that I need, and for me that means not hiding away the Sanity modules inside some nested folders where they don't really belong, just to get the behaviour that I want.

Naturally, I can only speak for myself. I'm sure there are many cases where such a compromise makes sense, or may even be the better solution.

krstcs
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Setting up Test Case dependencies

Post by krstcs » Tue May 02, 2017 9:54 pm

Yeah, I completely understand. I actually use a DB and stored procedures to dynamically control which test cases get run and what data is passed to them, so my setup is pretty complex. But not many want to go the route of a full DB and a front-end data management system to accomplish it (fortunately the let me do it here and I have the background to make it work).

Hopefully future features will help us both out. :D
Shortcuts usually aren't...