Automation: scope and purpose

Experiences, small talk, and other automation gossip.
User avatar
slavikf
Posts: 104
Joined: Mon Sep 13, 2010 9:07 pm
Location: Toronto, Canada
Contact:

Automation: scope and purpose

Post by slavikf » Sat Feb 12, 2011 1:08 am

I was discussing scope of Automation with some of our Software engineers (developers) and we can't come to conclusion... So want to ask colleagues, what do you think?

One opinion: main purpose of Software testing Automation is regression testing. So, that every time we build new version - we test, that all tested functionality continues to work, as expected...
Second opinion: main purpose of Software testing Automation (in addition to regression) is "smart" functionality testing. And "smart" testing is testing, in which execution path in not the same every time.

Take for example application, where i enter data in some forms, and then process and save that data in DB.
Now, as example of "regression" testing (1st case) i would take few sets of data (hard coded or stored in Excel file...) and will verify, that expected data shows up in DB.
Example of "smart" testing (2nd case) would be generating "random" data, which entered to forms and then verifying that we got appropriate data in DB.

Both ways can be done.
My question is hows generally testing is done?
Do you usually work on predefined sets of data?
Does your automated steps repeat every time same way?

User avatar
artur_gadomski
Posts: 207
Joined: Mon Jul 19, 2010 6:55 am
Location: Copenhagen, Denmark
Contact:

Re: Automation: scope and purpose

Post by artur_gadomski » Mon Feb 14, 2011 8:18 am

I think Automated testing is best applied to regression testing, that also includes smoke testing. Other great place to use automation are monotonous and repetitive tasks. Then there's something called Automated exploratory testing where you generate input data randomly and use some sort of oracle to evaluate the result. Problem is that regression testing is present in every/most project and the latter two are rather specific.
In my company we use automation for installation test and driver test (testing that every new driver has same expected functionality), we are currently developing Smoke tests and load tests, and plan for language tests.

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Automation: scope and purpose

Post by Ciege » Mon Feb 14, 2011 5:47 pm

Yes, yes and so much more...

I think typically companies start out with regression automation as it seems most logical. Continues verification that what once worked still works regardless of what changes or new features have been added to the AUT.

"Smart testing" as you put it is a little more advanced in that it requires more of a seasoned automation engineer to be able to architect and maintain a testing structure that can grow and change with your AUT. A lot of times these type of automated tests are "throw away" since one cannot always be certain that the AUT's features and structure will remain the same in the near or far future.

There is also stress and load automation testing that can, obviously, be designed to test the limits of your AUT. They (as well as the functional tests above) can also be used to test your AUT in critical system environments such as low memory, low hard drive space, low or no bandwidth, etc...

Automation also comes in handy with distributes installation and testing of an AUT that works in a connected environment.

So, I think basically, it's all a matter of what is most important to you and your company at the time of inception and what your team is capable of doing. Not all solutions fit all environments but with a little planning on the outset and determining what your scope is for the initial push you should be able to determine what will get you the quickest and best bang for the buck (ROI) then expand from there. So start small, then determine what direction you want to go in.

Clear as mud? If it were easy, then everyone would be doing it!
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...