Are Your Testing Metrics Misleading You?

If you are like me, there is nothing better than loading up the car with your favorite beverages and snacks and hitting the wide open road. Nothing compares with rolling down country highways and roads, good music playing while watching everything unfold in the rear-view mirror and the back windows.

Too much sun can come in the windshield (windscreen) and front windows, so I like to keep them covered and use the mirrors and sometimes take a look out the back window. That is the best way to know where you are going on a road trip, right?

I mean, you have speed and mileage indicators, and a fuel level indicator. It is even easier now with GPS either built in or on your mobile device. What else do you need?

If this does not sound like a good idea when driving, why do so many people manage testing efforts the same way?

Indicators

Several “indicators” are mentioned above. Speed has a speedometer. Mileage has the odometer. Fuel obviously has a fuel gauge. I expect an electric vehicle has something comparable, but I’ve never driven one so that is merely a guess.

Indicators are important. They can tell us a great deal about what is going on. They can tell us the rate of fuel usage, speed and distance travelled. An on-board compass can tell us what direction we are currently travelling. A GPS can give us a path to get to where we need to go.

All of these things can add up to something along the lines of “that is what we need.” It may be.

I have had engagements where clients or their “partners” had extremely detailed test plans and cases prepared. These were the roadmaps for the test effort. These were the guides to be used. They tracked progress by counting the test cases executed. I remember one that counted the number of test steps executed. These were executed per day.

Maps

A focused, alert vehicle operator will do far more than focus on the roadmap, or even the GPS. She will find information from the indicators helpful for shaping her thinking and progress toward her destination. Still, that is not all that she will do.
Her reference points from the map or GPS can give her information, so too can the speedometer and fuel gauge. To make sure she reaches her destination safely, she does far more than use the information items.

She will view the road ahead, shifting focus near and far, side to side. She will be alert to potential risks and stay aware as she drives the vehicle. A stopped or slowed vehicle on the side of the road, or even in her traffic lane is a reference point not contained elsewhere. Animals on the road or wildlife crossing the road need avoiding. Other obstructions, from branches, rubbish or even entire trees also need navigating.

Miles or hours driven can be easily measured. Test cases or test steps executed can be easily measured. With both we can “measure progress.” We might be able to estimate time of arrival or delivery. We might attempt to extrapolate when we arrive, or deliver software, based on how long we have been on the road, or testing.

We are estimating based on an estimate. We then presume the rate of progress will be consistent based on what we have done thus far. The correct term for this is “guessing.”

Looking Out the Window

If we are not actively engaged in what is around us, we will absolutely miss things. We will not see the lovely stand of trees. We will not notice the odd flash on the screen that is there and gone. We might not notice the group of deer by the side of the road, until they try to cross it. We might not notice the wrong value presented on the screen.

Measures and indicators can only tell us what we have seen so far. They are trailing measures, at best.

We must be aware of where we are going. We must be aware of why we are going there. We must look out the window to see what is around us.

The simple path defined by maps and detailed test plans will probably get us to where we are going. The way we get there is another question. By strictly following the easily measured test scripts, or the route provided by GPS, we will definitely get to an end point.

The typical testing metrics used to show “progress” show us what we have done, based on what we believed we needed to do before we started. They can show us the number of interesting things, anomalies, defects or bugs we discovered along the way.  They can tell us something about how our test automation is performing. None of them can accurately predict what is about to happen.

If we restrict our test efforts to the pre-planned scripts, we will miss issues customers, who are not following our path, will encounter. Limiting the number of interesting side trips for the sake of not impacting the performative metrics in use, e.g., test steps or cases executed per day or week, test steps or cases remaining to execute, will give an inaccurate view of the state of the software product.

If we allow for detours and interesting stops along the way, things might take longer than the simple, straight path would take. Taking side trips from the path is often where interesting and fascinating things are discovered. I have found this to be true for road trips, as well as testing.