How to Start Testing in Production

Why should you test in production? Bluntly, the truth is you’re already doing it; the true question is whether to become deliberate, expert and reliable at it, or whether it continues to be an accident and a source of crisis.

You’re already testing in production in at least the sense that you have procedures for handling incidents and outages that affect customers directly. With a little more investment, that workflow can become an ongoing, manageable asset for your software development lifecycle (SDLC). Here’s how.

Definitions First

For current purposes, testing in production has to do with web applications; the story for embedded software, installable software, and so on is outside the scope of this introduction. Also, TiP isn’t a trick that requires customers to find a mass of defects that the publisher should have found first. Most testing is still done as early in the SDLC as possible. Instead, TiP complements all other forms of testing for a few situations where the production environment is effectively irreplaceable.

TiP brings several situations impossible or impractical to match in a staging environment:

• Delivery at extreme scale
• Practice at error recovery
• Data which respects customer privacy while representing customer diversity
• Good return on investment

Highlights of these four themes follow.

TiP for Scale

Some applications are simply big, and to replicate their facilities or loads would be prohibitively expensive. It might be impractical to maintain more than one global network topology, for instance.

In principle, most development organizations maintain a staging environment where a product can be tested thoroughly before final deployment. But in practice, essentially all staging environments are smaller than the production environment in at least one dimension. This means that, even when all tests pass in the staging environment, installation in production introduces new challenges: an order of magnitude more co-operating nodes, hundreds of consecutive hours of traffic to shuffle caches far more thoroughly, a longer tail of unusual data, a thread pool that’s a multiple of the size of the staging environment’s, entirely different query optimizations, and so on.

The staging environment becomes a model for production, then: a site to verify that the application behaves predictably under modest stress. Full-blown execution only happens in production, though.

Business Continuity

One of the benefits of a focus on TiP is practice in useful disaster recovery (DR) of business continuity techniques. A good TiP team knows how to diagnose defects quickly, judge the best remedies for them, and, when the situation calls for it, revert back to a known good release reliably. TiP is an unmatched opportunity to verify an organization’s digital resiliency.

Real Data

Another aspect of TiP’s usefulness is the way it supplies test data. Conventional testing uses data that is similar to customer data, but of course not too similar, because that would violate the security and privacy of individual customers.

Production data, though, doesn’t just approximate the target; they are the target. Generating realistic data is a surprisingly expensive chore, and to be able to use real data correctly sometimes is a big advantage.

At the same time, rigorous controls need to be in place so that the data doesn’t leak out. For a programmer to accidentally disclose that “test-password” accesses a test database isn’t so bad; to share personal details about even one real customer, though, is a grave matter — and in many jurisdictions, an illegal one.

ROI

Engineering is always about choices — that is, finding the right balance between costs and benefits. TiP shouldn’t be seen as a last resort only for those situations when no other test is possible. Instead, think in terms of situations where TiP pays off.

It might be feasible, for instance, to create test data that adequately models customer data for a specific load test. If running such a test as TiP liberates experienced analysts from data synthesis so they can focus on analyses that directly boost revenue, then TiP immediately becomes profitable.

Tips to Make the Most of TiP

The first and most important step toward TiP is the flexibility to consider it. Even with that in place, a great deal of technique remains. In fact, TiP has grown so big and beneficial that it’s a specialty unto itself. In particular, TiP techniques in the three phases of deployment, release and post-release are arguably as refined and complex as all of classical pre-production software testing. Comprehensive treatment of this range of methods is far beyond the scope of this introduction.

A few pervasive tips about TiP are worth recording, though:

Work with feature toggles to make them second nature in your designs and implementations

This and all the tips which follow promote the strategic role of TiP. Feature toggles (developers often call them “feature flags”) are handy, for example, in analytic efforts, but they bring advantages far beyond this. They promote rapid time-to-market and agility. They often decouple complicated linkages. Feature toggles make schedules for deployment, testing, and feature delivery more independent. The consequence: deployment’s calendar can minimize risk, testing’s schedule fits employee schedules, and feature delivery is free to make the most of market opportunities. Trustworthy feature toggles make trade-offs between these different domains far easier to manage.

In all these ways, feature toggles help make TiP not just a technique for coping with resource constraint, but a strategy which enhances functional teamwork and shortens the time to deliver value to customers.

Schedule slack resources so you can focus your first TiP efforts on the times and places your application is least loaded

TiP is risky, to be sure, but a good plan can wring out most of the risk. Make things easy on yourself: schedule TiP deliberately when customer demands are light and plenty of testing and engineering staff are on hand. Experiment with simple TiP, and build up from their. Keep practicing until TiP becomes so routine that the staff knows what to expect, and all the fear is gone.

Instrument the application so you can see what customers see

Responsiveness to customer complaints is a good intermediate goal. A still more evolved goal, though, is to anticipate complaints: build in alarms and monitors that put the spotlight on problems even before customers realize they’ve happened. Every time TiP seems hard, turn that perception into an automation or dashboard that helps the engineering organization stay ahead of customers.

Create a variety of auxiliary test accounts that obey the same rules as govern other customers

Part of vision into customers’ experience is to experience their limits. Customers inevitably have limits on their account. Make sure that testing explicitly addresses all those limits, including number of transactions in a day, ability to view the information of others, and so on.

Enlist a security specialist to review your TiP for hazards and insights

Security is hard to get right. Security for TiP is a particularly new arena. Find someone who can take responsibility for the security of your TiP efforts, and listen to the insights that emerge.

By using these techniques, you can make TiP a manageable asset in your testing arsenal, and succeed with TiP.