Is It Safe to Tell Continuous Testing Your Secrets?

At the risk of giving away the answer to the question in the title too early, no: In general, secrets should not go to continuous testing (CT). While the production version of your application needs a good secrets solution, it’s probably best to segregate that from your CT.

The full story goes far beyond a one-word answer, though. We need to truly think about secrets, what they mean and how to handle them.

Caution: Education in Progress

First, recognize that we’re still only in the infancy of computing. It’s effectively 1910 in aviation, when all sorts of strange dead ends showed up in the skies.

While there were reasons for all these experiments, a lot of them turned out not to be worth the trouble involved in constructing them.

Similarly, computing remains hard in 2020 largely because software involves so many choices, and specialists still have to think through so many of them. Should a database password show up in a CT configuration? In isolation, that sounds like a bad idea; as we’ll see below, though, sometimes it’s warranted.

Keep in mind that someday we might all recognize that some of these possibilities are the equivalent of a seven-winged aircraft: not the way to go for the long term.

What’s a Secret?

For this purpose, a “secret” is anything that shouldn’t appear in public: a token to authenticate against a computational load balancer, the hostname for your primary NAS, even the account IDs for your three largest customers or the name of the city where your CFO lives.

Presumably, your CT isn’t public, but CT practice is also juvenile enough that the one or two firewalls that might protect it aren’t enough for serious security. The risk that others might break into your CT is too great to be casual about it.

Consider these four alternatives in the specific case of a database connection.

Mock

Quite a few unit tests can be effectively mocked. It’s straightforward, for instance, to write good mocks to exercise a database’s exception-handler for a network outage, wrong password or inadequate permissions.

Mocks can be entirely public; they generally don’t have to disclose any secrets at all. Mocks also typically are fast — nearly always faster than the resources they mock — and most tutorials on testing treat mocking as necessary.

Mocks cost programmers time and attention, though. They often are harder to write and maintain than the code they’re intended to help test. Mocks ultimately leave an empty feeling, in that a true end-to-end test picks up conditions no mock can touch, such as an inconsistent schema.

In-environment database

In some situations, especially when the database technology has a permissive license, it’s feasible to set up a database server in the CT environment. This obviates dependence on mocks and makes the construction of new tests far easier than when mocks are required.

Work to bring up a database in your CT environment probably applies directly to disaster recovery or business continuity efforts, and vice versa, so construction of an in-environment database might bring the benefits of two projects, a little more than the cost of one; that has the potential to be a good thing.

Passwords in such a construction can be specific to CT. Publishing them is no hazard because they have no meaning outside CT.

The biggest limitation of an in-environment database is unlikely to be any particular functionality of the database. Robust deployment methods minimize any difference in capability between the ephemeral CT-based DB and the persistent production database. For some applications, the differences between a test and a production database are inconsequential. The backing store for an application with the little state, for example, likely exhibits all the same behaviors whether the database is in the CT or on production servers.

However, problems are likely to turn up with unknowns, such as if the production database has been customized in an undocumented way.

Staging database

A third distinct possibility for the model problem of managing database-related secrets in CT is to create a long-lived staging database.

This can be nearly an exact replica of the production database, with the likely differences being that the staging database:

• Is on a less rigorously guarded platform
• Probably has less capacity in size and speed
• Contains test data rather than production data
• Potentially is available as a resource for other development activities beside CT

Applications that need to test scale find advantages with staging databases, as compared to CT-hosted databases. The latter might practically load tens or hundreds of megabytes of test data, while a staging database can easily have tens of gigabytes or more of data. The latter is only managed practically in a configuration where they can be persisted from one CT run to the next.

Note, though, that some CT environments are gaining the ability to attach external or persistent resources. Eventually, it might become commonplace for “the staging environment” to be hosted by the CT platform.

Production database

Testing in production?! That’s surely just a hallucination, right? So continuous testing with production resources must then be doubly unhinged.

Not necessarily. Leading database technologies make it easy to duplicate the schema of the production database with the same server but in a distinct database. The second, test database might be accessible only by a special test user, whose password is of low value.

The advantage of such a choice is that it opens up more of a truly end-to-end test for CT. In this arrangement, CT connects directly to the production database, just as the usual application does. It runs against production database functionality. It experiences the same network connections, schema and filesystem weather as the production database does, while the data remain segregated. For testing applications with sensitive dependence on their databases, those might be important advantages.

Manage Your Secrets in CT

Commercial software appears to be evolving toward standards for secret management that will solve the current difficulties CT faces. Until that happens, though — until your organization adopts a trustworthy vault that consistently works across development, testing and production needs — keep in mind that you still have a range of options for keeping secrets from leaking through CT.