5 Steps to Succeed with Testing in Production

Does “testing in production” (TiP) sound like crazy talk? Does the thought of working right next to customer data sound as comfortable to you as cooking dinner next to a crocodile pool?

Those are healthy instincts. But testing in this manner doesn’t need to be a risk-filled activity. Here’s what you need to know so that TiP becomes not just possible, but sensible.

I earlier wrote about reasons to test in a production environment and how to make the most of it. As I mentioned there, though, TiP is a rich subject and deserves deep study.

Still, there are definite ways to set yourself up for success. (Note that the points here apply best to TiP for software as a service (SaaS); while installed and embedded software have their own related TiP concepts, they’re different enough to treat separately.)

Take these five basic steps now to make sure you’re well equipped for your TiP journey.

1. Know your load cycle

The ideal for TiP is to do it so carefully and perfectly that it’s safe. Computing systems always change, though; if only for that reason, wise engineers plan on imperfection.

One way to do that involves knowledge of your production systems’ habits through time. Does your load go down on the weekend? Does it go up then? How do your customers’ needs vary over a 24-hour day? Do holidays or weather events consistently shift demand for your service?

Awareness of customer habits helps you find favorable times on the calendar for testing. You’ll want to run many tests when other demand is light. Certain kinds of user tests, though, go best when spread over as many distinct users as possible, for as short a time as possible.

You’ll be in the best shape to manage these schedules when you have clear and accurate details about what your customers do.

2. See what your customers see

Provide testers and support staff with whatever is needed to see faithful representations of what customers see. To make workers guess or ask clients how things look on their side just introduces noise, confusion, and delay.

Make sure they can access the browsers, hardware or remote-control facilities, or other resources needed to share views meaningfully. If privacy is a constraint, build in a way to experience what clients experience, except with crucial details blurred or anonymized. Looking at the same picture is crucial for understanding customers’ experience.

3. Get security help

Security considerations change rapidly. Your service needs a security specialist both to review the security consequences of your TiP plans and to advise about security tests to integrate into your TiP program. Find a way to adopt DevSecOps, or at least bring in someone who can be trusted to ensure your TiP doesn’t harm your security profile.

4. Use multiple test accounts to model different customer profiles

Left to themselves, many developers and even testers default to the use of personal accounts with elevated privileges for testing. That’s not good enough. Testing at least part of the time with customer-level accounts is essential.

Don’t leave this to chance. Explicitly construct test accounts that model the personas your marketing department has defined or your support staff has profiled. While use of a personal account that’s reconfigured has its place, don’t rely on that for your formal test plan; assume that testers need to manage a variety of different accounts with well-defined properties.

If your system doesn’t support “extra” test accounts, that’s a problem that must be solved. Exercising a system with a variety of profiles and permissions is fundamental to testability, however many organizations neglect it.

5. Learn TiP’s fundamental techniques

To teach all that’s necessary for TiP is beyond the scope of this introduction. The point for now is that TiP is distinct from more conventional testing. Your study of TiP will make much more sense once you clearly understand some fundamental techniques:

• A/B testing
• Canarying
• Distributed tracing
• Feature flagging
• User engagement testing

Learn these techniques in whatever way makes sense in your situation: sit in a college classroom, read some books, write your own implementation, attend a conference, organize a talk series for your team, or whatever is effective for you. To progress with TiP, you need to take advantage of those who know more, and it’s easiest to understand them when you share a basic vocabulary.

Make TiP a strength

If TiP seems risky or intimidating, remember: You’re already testing in production. With a little care, you can do it well: manage your plans and results, enhance security and stability, and make TiP a strength rather than an accident.

Recognize that testing is always a team sport. Even this brief introduction to TiP gives ways to align with colleagues in marketing, support, development and infrastructure. You go farther and better when you learn from others, and especially when you learn the values they see in TiP.