Cleanup after failing tests

Experiences, small talk, and other automation gossip.
boegvald
Posts: 17
Joined: Wed Mar 13, 2019 9:02 am

Cleanup after failing tests

Post by boegvald » Fri Aug 09, 2019 12:32 pm

How do you cleanup automatically after failing (browser-based) tests?

This theme seems to be an endless one for me. A test can fail in so many ways, leaving browsers open with open modal dialogues, file dialogues, etc. For now, I have ended up with starting all tests with trying to close all existing browsers, then killing any browser processes left. But some times that leaves some browsers (like Chrome) to start up with warnings, which then have to be handled… It feels like a never ending challenge…

And yes, I know, tests should always leave the environment intact. But they doesn't ;-)

Do you have an effective way of cleaning up? Are you using some kind of sandboxing or virtualization to start from scratch each time?

All good ideas welcomed…

(Primarily doing browser based tests, stored In GitLab, run on a virtual server via Jenkins).

Best Regards
Bo

User avatar
RobinHood42
Posts: 298
Joined: Fri Jan 09, 2015 3:24 pm

Re: Cleanup after failing tests

Post by RobinHood42 » Tue Aug 13, 2019 1:53 pm

Hi,

There is no sandboxing or virtualization in Ranorex. It simply tests your UI. If your test ends up with open browsers I assume the test doesn't work correctly :wink:
There seem to be some edge cases you didn't implement in your test scenario. If you know which dialogs are open, you can close them the teardown section.

Cheers,
Robin :mrgreen:

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 4967
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Cleanup after failing tests

Post by odklizec » Tue Aug 13, 2019 3:08 pm

Hi,

One thing you can implement, and which I'm successfully using for one of my TA projects, is creating a VM snapshot, just before the test starts, run the test and then, regardless of test outcome (success of failure), return to the snapshot state. And because the snapshots could become quite big and clutter the VM storage, always delete the snapshot. This approach could help you with opened dialogs, improperly closed windows and changes done in system. What it will not help you with, are potential DB changes, unless the DB is stored locally. Otherwise, you must find a way to restore DB back to its previous state. Hope this helps a bit?
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

boegvald
Posts: 17
Joined: Wed Mar 13, 2019 9:02 am

Re: Cleanup after failing tests

Post by boegvald » Tue Aug 13, 2019 3:49 pm

Good to see that you have been successful at using snapshots to 'reset' the testing environment. Which technologies are you using?

I was considering a variant of your setup, creating a snapshot once, then starting each test with returning to the snapshot. But your approach seems somewhat more flexible (systems updates and other environment changes are easier to handle).

But I can find no information about using Jenkins to work with Hyper-v snapshots… Maybe there is a plug-in I just can't find? Or it's just a matter of writing a couple of power shell scripts? Maybe not?

Restoring the DB is no problem as I already have a way of doing that.

Best Regards
Bo

User avatar
odklizec
Ranorex Guru
Ranorex Guru
Posts: 4967
Joined: Mon Aug 13, 2012 9:54 am
Location: Zilina, Slovakia

Re: Cleanup after failing tests

Post by odklizec » Wed Aug 14, 2019 9:39 am

Hi,

I'm using vSphere plugin in Jenkins, which is configured to perform all snapshot-related steps. Unfortunately, I have no experience with Hyper-v. If there is no Hyper-v plugin in Jenkins, then you may need to use command line/powershell to create/restore Hyper-v snapshots? Check for example this blog post:
https://devblogs.microsoft.com/scriptin ... owershell/
And also this one:
http://hyper-v-backup.backupchain.com/h ... snapshots/
Hope this helps a bit? ;)
Pavel Kudrys
Ranorex explorer at Descartes Systems

Please add these details to your questions:
  • Ranorex Snapshot. Learn how to create one >here<
  • Ranorex xPath of problematic element(s)
  • Ranorex version
  • OS version
  • HW configuration

boegvald
Posts: 17
Joined: Wed Mar 13, 2019 9:02 am

Re: Cleanup after failing tests

Post by boegvald » Wed Aug 14, 2019 11:54 am

Thanks much for the 'hints'. It doesn't look too complex. I'll see if we can give a try :-)

Best Regards
Bo