Global test case timeout failure

Ranorex Spy, Recorder, and Studio.
cruzez
Posts: 24
Joined: Thu Jan 27, 2011 3:03 am
Location: London, UK

Global test case timeout failure

Post by cruzez » Mon Jun 02, 2014 4:08 pm

Hi,
Is there anyway I can set global timeout for each test case to failed if it takes beyond certain time for ex 3 minutes? Perhaps some settings I can set globally?

Normally my test cases are at modular level, each test case never takes beyond 2 minutes. I want a test to be hard fail it reaches deadline timing.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Global test case timeout failure

Post by Support Team » Wed Jun 04, 2014 4:03 pm

Hi,

May I ask you to describe your issue in more detail?
Could you describe what shouldn't take longer than 3 minutes, the whole test case or the time Ranorex searches for an elements?

Thanks,
Markus
.
Image

rastek
Posts: 157
Joined: Wed Aug 06, 2014 12:00 pm

Re: Global test case timeout failure

Post by rastek » Fri Oct 24, 2014 8:02 am

Hi
I also have problems with global and level timeouts.

I want to decrease timeout for particular objects, I both tried from GUI and from code but failed to decrease .

I see there is only top-level application timeout changing possible.or maybe I am missing something here but I follow support forum for these but could not succeed

Also I did not understand what is timeout in settings used for.When I change Search Timeout from repository Item properties page, it just adds it to Effective Timeout.

When I change all three timeouts from Settings->Repository Defaults->Timeouts for new events
again nothing changed.

By the way what is 'Timeouts for new events' that means not changing items already in repository ?

I also tried record item again after changing this but again nothing changed.

I dont understand why it is so hard to change timeouts in the Ranorex.

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Global test case timeout failure

Post by krstcs » Fri Oct 24, 2014 2:02 pm

The timeout setting in the Ranorex settings dialog is the default value used when you create a NEW repository item.


The effective timeout is the sum of the element's timeout plus all parent elements' timeouts. This cannot be changed directly. If you need to change the effective timeout you will need to adjust the timeout of the element and all parent repo objects.

For example, the default timeout for Ranorex is 30s. Let's say I have the following repository items:

MyWebPage => /dom[@domain='www.mydomain.com'] (timeout = 30s, effective timeout = 30s (no parent))
--MyDivArea => //div[@id='MyDiv'] (timeout = 30s, effective timeout = 1m (MyWebPage timeout + MyDivArea timeout))
----MyATag => /a[@id='MyATag'] (timeout = 30s, effective timeout = 1.5m (MyWebPage + MyDivArea + MyATag))


If you want to change the effective timeout of MyATag to be less than 1m, you would need to adjust MyWebPage and/or MyDivArea and/or MyATag.

It is done this way so that each level can be controlled separately.

Note: 30s is normally a good timeout and should not need to be changed. Remember that the timeout only tells Ranorex how long to wait before it fails. If Ranorex finds the element more quickly then it will continue as soon as it finds the element.
Shortcuts usually aren't...

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Global test case timeout failure

Post by jasoncleo » Wed Aug 12, 2015 6:44 am

Apologies for resurrecting an old thread, but I have the same question as the original poster, and their question was never answered.

The question they have (and that I also have) if we have a number of test suites to run, and they should normally only take X minutes to fully execute, then is there anyway that I can set a config value so that Ranorex fails the test suite if it goes over a threshold time value?

So for example, TestSuiteA.exe should only take 15min to reasonably get through all its test cases and iterations, but I would like to also be able to set a config value so that if it exceeds 30min, then that process aborts with Ranorex marking it as a failure.

This would be in the event of:
- Maybe bad coding update in the test case design that gets stuck in a poorly written loop
- Some excessively poor response times from the AUT
- Some alignment of the planets that has the test case stuck in some weird race condition

I understand the argument that the user code modules should be written well enough for nothing like the above to happen, but at the end of the day, the guys churning out our Ranorex test case code are all QA staff, and not hardcore developers so they are prone to occassionally make mistakes.

So: If there some test suite commandline parameter to define a hard time limit for execution?

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Global test case timeout failure

Post by krstcs » Wed Aug 12, 2015 2:04 pm

No, there is no such parameter built-in.

But, what you can do is create a couple of modules that you put at the start and end of the test that validates the running time.

The first module would be put in the test suite's [SETUP] area (right click on the test suite and select "Always show SETUP/TEARDOWN". It would grab the current time and save it in a module variable that is mapped to a global parameter.

The second module would be placed in the test suite's [TEARDOWN] section and would take the global parameter from above and subtract from the now current time at the end of the test and validate that that duration is less than the expected value.

This won't help stop the test if it runs long, but it will at least cause the test to fail if it goes long. You could put these modules in each test case's SETUP/TEARDOWN and set the timeout for each TC instead of the whole suite.

I would question why you are worried about how long it take, given that this is a functional test automation tool. Is there a software requirement on the SUT that dictates that the process take no longer than X amount of time? Is it an environment issue where you only have the test system for X time? There may be other ways to handle these types of issues that won't make the tests fail just because they go longer than desired.
Shortcuts usually aren't...

User avatar
jasoncleo
Posts: 37
Joined: Mon Jun 08, 2015 7:37 am

Re: Global test case timeout failure

Post by jasoncleo » Tue Oct 13, 2015 4:00 am

Sorry for the lag in getting back to you. I haven't checked into the forums for a while.

The requirement for this was in relation to Continuous Integration. We have a selection of our tests that run at the end of a Jenkins pipeline.

However CI being what it is, means that while the tests are running, it is then keeping Jenkins from running other stuff. Yes, I know you can set up multiple-executors, but that is a debate that I'm not going to detail here as it is more complicated than it seems with Jenkins, and our current infrastructure limits the number of VMs that IT is going to let the test team have, so we couldn't get a dedicated Jenkins agent for test.

We just want to avoid the case of tests running for 8 hours, when we would expect that the suite should be done in 5. (We have a pretty big project so the sanity suite has grown).

In the end, we ended up having Jenkins enforce the timeout and I wrote a utility that manages and tracks our various test environments to notify people when they had test suites aborted by Jenkins.