Design, Build, and Implement a Continuous Testing Process

If your organization has decided to practice continuous testing (CT), what’s the next step? What do you need to know?

For one thing, it costs less to begin than you might think. Continuous testing accommodates a small start, along with incremental growth. You should plan a useful “live” CT run in the first week of operation — maybe even on the first day.

To launch your CT initiative successfully, understand what practitioners generally mean by CT. The focus is not on testing without cessation or interruption, as much as the name suggests that. Instead, CT is generally considered in the context of a software development lifecycle (SDLC) where developers commit source code at various times.

The usual interpretation of CT is that each event defined by submission of a changeset triggers a testing cycle. If those tests fail, then the changeset is rejected, and the application reverts to its last accepted state. If we’re being literal, the practice may better be called low-latency, event-driven, or fast-feedback testing, rather than continuous testing.

One variation on this theme calls for testing to occur on a definite schedule, except only as needed. An organization might decide, for instance, that the repository be scanned every two hours during the work week; if an update appears, then a test cycle begins.

Design: start with events and actions

An even more important refinement of the CT intent is to define different scales of testing. For instance, an organization might schedule these three scales:

• Every minute, a collection of quick regression unit tests fires
• Every time a commit is made, certain relatively fast integration tests are launched, and the commit is reverted if any exception turns up
• Every day at half-past midnight, a collection of time-consuming performance, security and integration tests runs

We design combinations like this to make the most of the expenses of testing. The payoff, of course, is to identify an error as close as possible to the introduction of that error. Notice that each of these three scales plays an irreplaceable role. It’s valuable, for example, to have tests that do not depend on developer commits: these ideally will be the first to detect an error in an external dependency.

Start a CT program, therefore, by diagramming available SDLC tooling–integrated development environment (IDE), source code control systems (SCCS), continuous integration (CI), test automation, and so on–and how the pieces best communicate.

Build: take an incremental approach

Clarity about these different possibilities encourages an incremental approach to CT. What’s your single most common or damaging category of error that slips through your organization’s existing workflow? That’s an ideal place to start.

If there have been occasional transient errors in authentication that show up inconsistently for different users, you can design a big exercise to run an appropriate minute-long, end-to-end exercise for each of a hundred users. Automate and launch that test every morning at 1 a.m. and results will be available well before 3 a.m. That becomes your first step in CT, and the team can build from there.

Or is it too common for programmers to check in code that looks right, and even passes peer review, but it ends up working only on specially configured hosts? Define the right unit test, iterate it over the full range of pertinent platforms, and block acceptance of any changeset until the regression unit test suite completes. All of them together shouldn’t take more than a few seconds.

Implement: give it bite

A first step need not be perfect or comprehensive; it’s more important to be meaningful, well understood, and easy to maintain. A good CT plan starts small, with early wins, and promotes extension and expansion.

It’s also essential that CT have “teeth” from its first run; because if CT results can be ignored, they will be. A failure of a CT test needs to have a real consequence — generally to block acceptance of a deliverable. CT can’t be “on the side” of your SDLC; it must be in-line. Establish CT as a standard operating procedure which the whole team always uses.

CT isn’t a product or commodity, and there’s no way (for now) just to order CT as a line item. For at least the next few years, someone with a specialized focus will need to tie together a repository, testing tools, test data, and probably a notification or ticketing system in order to implement a useful CT installation.

Automate: shorten cycles with test automation

Notice how CT shifts the role of test automation in SDLC. CT triggers test events “to the left”, soon after source code exists. Automation promotes CT’s ability to schedule tests early; automated tests identify problems within seconds or at worst hours of their creation, when they are often easier to debug and cheaper to fix.

Make the most of opportunities for CT

Two special circumstances deserve consideration in any CT plan.

First, look for places throughout the SDLC to apply CT. While it’s most common to apply CT in the vicinity of code check-in, other SDLC points are possible. A product with a complex generation procedure might define CT with unit tests on source commit, then a later stage of CT to validate generated products.

Second, It’s also possible, at least in principle, to have CT without full automation. As one aspect of incremental management, it’s sometimes wise to define a CT “gate” before the gate is fully mechanized. A plan might call for verification of a human-readable result of a complex operation, for instance. It’s OK to stub one CT step so that a trusted testing specialist views a result and pushes a button to accept or reject it. While such a step should be automated as soon as possible, it can be valuable to make the test part of a larger routine from early on, even before full automation.

In summary:

• plan your CT approach ahead of time
• start small with easy wins
• make sure you can expand your plan later

Be sensitive to different scales of CT, particularly across time ranges and platforms. Look for the errors that bring the greatest business risk and figure out how to create CT that manages that risk. And recognize that CT itself likely will require daily maintenance and refinement, but that daily effort pays off in a stronger overall test strategy.