How do you handle (SUT) bugs & crashes in your test scripts?

Experiences, small talk, and other automation gossip.
oroosen
Posts: 3
Joined: Fri Jul 27, 2012 7:25 am

How do you handle (SUT) bugs & crashes in your test scripts?

Post by oroosen » Mon Sep 10, 2012 1:09 pm

Hi, while writing a script for a test, the SUT crashed. This got me thinking "what should I do with my test script until the bug is fixed? Shall I drop it, shall I run the script and crash? ".

I'm thinking about running the script and querying the bug tracking system. Then I will report an error, either the bug isn't fixed or my test script must be updated. I'm not sure what to do with a crash. Restarting the SUT allows me to run more test scripts.

I'm curious how others deal with such situations.

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: How do you handle (SUT) bugs & crashes in your test scripts?

Post by Ciege » Mon Sep 10, 2012 5:52 pm

What I do if I know I have a blocking defect is the following...

1) write a snippet of code that checks the status of a defect (you can do this if you have some sort of access to your defect database, otherwise you can skip this step).
2) If the defect is not marked fixed, or if you are skipping step #1, write a WARNING or a FAILURE to the log file that the following bit of test code will be skipped until defect #XXX is fixed.
3) skip that section of test and continue on, assuming that the defect will not affect the next set of tests.


The thinking is, if I just stop testing there, and there still are tests that can be run and are not affected by the defect, then I would be missing on importing testing in other areas of the AUT. I do not want to skip testing when it could still be accomplished.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

oroosen
Posts: 3
Joined: Fri Jul 27, 2012 7:25 am

Re: How do you handle (SUT) bugs & crashes in your test scripts?

Post by oroosen » Thu Sep 13, 2012 10:13 am

How often do you check if the SUT has crashed? Or handle validation errors?
They both leave the SUT in a state from where it's difficult to continue. Do you kill the SUT, restart it & continue?

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: How do you handle (SUT) bugs & crashes in your test scripts?

Post by Ciege » Thu Sep 13, 2012 3:45 pm

How often do you check if the SUT has crashed?
I have a running second thread that checks for any unhandled exceptions in my AUT. This thread will halt testing when one occurs. Since it is an unhandled exception in the AUT I want to preserve state and know my latest steps so that I can get it addressed.
Or handle validation errors? They both leave the SUT in a state from where it's difficult to continue. Do you kill the SUT, restart it & continue?
Depends on the error... You have to know your AUT and your test code. You also have to know what expected values are and if an incorrect value will affect your future tests. If an incorrect value will affect a future test you need to fail your test, put your AUT back in a known good state and continue testing. Or, just fail your test and stop testing so you can manually verify the failure and determine the error.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...