How Much Testing is Enough?

One common question on software development teams is “When should we stop testing?” That simple question has loads of ways to answer that. Books and numerous papers have been written and many presentations, lectures, and workshops around answering that question.

Recently, Ranorex partnered with Pulse Research to ask IT leaders how they decided when to stop testing. Here is what they had to say:

Asking the Right Questions

Peter G. Walen, conference speaker and member of the Agile Alliance, the Scrum Alliance, and the American Society for Quality (ASQ), shared with us a different perspective on the question, “when to stop testing.” Here is what Pete had to say:

Recently, I was asked a question that was different. The team had a variety of tests, each exercising interesting aspects of the application. They had progressed from simple checks against requirements to testing around requirements. They were doing deep dives into areas that caused them problems in the past.

They were looking at areas of concern not covered in “requirements” or “acceptance criteria” but their experience with the product told them these were likely trouble spots. They also used their knowledge of how their customers used the software and expected it to behave.

They had a mix of manual and automated tests defined. They had multiple levels of complexity covered by the tests, depending on what they intended to find out about the system. Some were simple “smoke tests” they could use in their CI environment. Some were more complex integration tests looking at interactions between segments of the system.

Their question was, “We have all these tests we regularly run. We are developing more tests as the product changes. How much testing is really enough?”

Step 1: Determine what the team means by “enough”

Asking a question about “enough testing” might be confusing to some organizations. They might understand “testing” that consists of positive confirmation of what is “expected” and not looking beyond the “happy path.” The idea of “one requirement: one test” is normal. The challenge is, while this may be acceptable for some organizations it is something less than that for many others.

Then there are other organizations, like the one I described in the introduction:

• They cover as many scenarios as possible.
• Tests that make sense are included in their CI test suite.
• Other tests are included in their automated integration and regression test suites.
• They are using tools to run tests that have provided interesting or unexpected results in the past, so their skilled analysts can focus on new work and not waste their time repeating steps that could be run by an automation tool.

Nearly every other organization is somewhere between these two extremes.

Step 2: Avoid the “not as much testing as we think” trap

Many organizations are closer to the minimal “one requirement: one test” than they realize. A simple script is created and intended to be an open-ended question. The steps are expected to be run several times with different values which can be correct or incorrect in various ways.

Testing around a requirement is what is expected with such a test. Except, when deadlines are looming or past, and massive pressure is being applied to “finish testing,” such tests might be run two or three times, instead of the seven or eight they might otherwise be executed. They might not exercise all the possible logic paths, even if they are aware of them.

Corners get cut for the sake of time. People who are looking only for the checkbox of “this test verifies this requirement” are likely not going to consider what “verifies” actually means or implies. They have fallen into the “not as much testing as we think” trap.

While the intent is there and we can recognize the goal, they are falling short of that goal.

Step 3: Avoid the “kitchen sink” trap

Some teams or organizations look for testing to cover every possible behavior and combination of values. They look for testers to evaluate everything possible in the system and fully document, or at least fully exercise those possibilities.

Then testers are expected to repeat the tests. All of them, for every release and every build.

The volume and amount of work needed to be done are overwhelming. Even if they try to automate tests, continuing to run every single functional test in the name of regression testing becomes impossible. Tests get left out. Tests that are quick and easy to run often are selected in place of more complex tests that take much longer.

Why? Because when the management team realizes the Herculean nature of the task, they often settle for some percentage of the tests being executed. If the test team can get 80% of the tests run by doing small simple tests, then they can focus time on the new features that need more careful thought.

Step 4: Find the right balance

What does the idea of “enough” mean? There is a balance between the two extremes. What and where that balance is depends on the situation.

I find a level of testing which allows for in-depth exploration of key features, along with reasonable coverage of secondary features to work much of the time. What will be tested more and less than other features should be discussed with the stakeholders and project team so everyone is in agreement. Then, regression and integration tests need to be updated accordingly to handle the changes.

Where these fall will vary by organization, team, and project. In short, the first team I described did a very good job finding their balance. It rarely happens on the first try and can take some effort and patience. It is worth it.