Which is best practice? To close or not?

Ask general questions here.
stapes
Posts: 206
Joined: Wed Sep 16, 2015 10:55 am

Which is best practice? To close or not?

Post by stapes » Wed Oct 14, 2015 10:42 am

Hi

I am testing 3 different versions of the same software, a Windows app, an Android App and an iOS App.

So far I have a battery of 15 identical Test Cases to run on all three.

None of the Apps have an internal (built-in) close command.

I am closing the apps from within Ranorex using this:

Code: Select all

Host .Local .CloseApplication (repo .AgileMobileApp .Self );
This works fine, but then subsequently takes about 30 seconds to restart the App for the next test.

On the positive side, Closing & re-opening the Application guarantees starting at the same, known starting point every time. The other way, a previous app could have failed & left the control anywhere.

Would it be considered Best Practice to close & open this app for each Test Case/Iteration?

User avatar
Aracknid
Posts: 388
Joined: Tue Aug 10, 2010 3:23 pm
Location: Toronto, Ontario, Canada

Re: Which is best practice? To close or not?

Post by Aracknid » Wed Oct 14, 2015 3:00 pm

Interesting question. The answer (IMO) is that it depends on what you are trying to accomplish.

Just so we are clear, I test Web browsers, not phone apps. But I think this doesn't matter to the question.

On one hand, starting from a known (clean) state for every test will allow you to run through a bunch of tests, over and over, testing specific functionality of the app under test. If you organize your runs in a smart way, you can test functionality from one test to another, building on the success of what you already know works. this allows for smaller, specific tests AND when something fails, it is probably easier to say what is working and what is not working in your app. It's probably more stable.

On the other hand, it wastes more time as you have to constantly re-run code over and over again to launch your app and navigate to where you want to be in your app, to get to the state of the start of the next test. Also, who really uses your app in the real world by starting it, doing one thing, stopping, restarting, etc...

It also depends on your testing environment setup. For example, how many computers (real or virtual, virtual likely) you have available to run tests, how many Ranorex licenses you have, etc... If you can only run one test at a time on one config at a time, then time might be something you don't want to waste. But if you have the resources to do many things at the same time, and to kick it off over night, then perhaps time is not an issue.

For me and what I test, I do clean starts and mostly individual tests that test specific functionality. It definitely wastes about 1 minute or so between tests to shut down the current test and then start the next, and then maybe a few seconds or minutes to navigate back to a state I need to be in, but I'd rather have 10 tests that test 10 specific things and look at the results for those 10 tests and know that 10 things work, then have 1 test say the same. Mostly for when things fail, this is important to me. If tests 1-6 pass, and then test 7 fails (and depending on if tests 8-10 are dependent on the run of 7 or not) I can quickly look at the results and know what is not working. If this is all 1 test, the result doesn't tell me anything, and I have to look at report/log files to figure out exactly what doesn't work. Then I have to re-run the entire thing again, which wastes time. Maybe test 8-10 also need to be re-run or not. I also have a whole bunch of virtual machines and servers that I manage for all this, and enough licenses that I can run many things at once, and a process to do this all with the click of a few buttons and overnight.

But I also often throw in a super test, to mimic more of a real world test. This test would perhaps test everything in one big test, stringing together elements of previously tested functionality. Most of the test is a repeat of what has already been tested, but it takes much longer and does much more in one flow.

Anyway, those are my thoughts.

Aracknid.

Martin
Posts: 152
Joined: Fri Aug 15, 2014 12:24 pm

Re: Which is best practice? To close or not?

Post by Martin » Thu Oct 15, 2015 9:39 am

Best practice would be to follow the needs of the testing.

Further more it's best to start a test case cleanly, meaning a clean start for the app itself. Depending on the need of the test case this can change but mostly these are specific cases.

Also, I'd consider the info you've got from re-launching the application and considering it as an issue. For user experience 30 sec start-up of an application is not that good and might reflect to some bottlenecks.

stapes
Posts: 206
Joined: Wed Sep 16, 2015 10:55 am

Re: Which is best practice? To close or not?

Post by stapes » Fri Oct 16, 2015 5:13 pm

Interesting thoughts.

I need not worry about that issue of the 30 second start up, as the application is not normally ever closed.

So in the real world, this will never happen.

That's why I said it has no internal Close command and have to close it from code.