Take Your Continuous Testing to the Next Level

You’ve got your continuous testing (CT) humming: programmers check in their code, and within 10 minutes out pops a preliminary report confirming that all unit tests still pass. Lovely! And it’s a big benefit, too, because on the rare occasions that a problem does turn up, it’s solved with far less effort because it’s identified quickly, before it becomes obscured by all the distractions that accumulate over hours and days of focus on other issues.

That’s just the way CT is supposed to work. What your team might not realize, though, is how much wider CT can spread, well beyond the programming language the organization identifies with your output.

Do you build your software with Bash or PowerShell or Make? Those scripts deserve testing, too. Does SQL mediate your database access? Boost the productivity of your database specialists with automated validations. Are your CSS sources accumulating the technical debts of dead specifications and deprecated archaicisms? Bring in an automatic tool to help clean out that dirt.

Testing that more closely models customer-like interaction, is the usual focus of testing technology, and, of course, work at that level will always be primary. Automated confirmation of specific artifacts has a lot to offer on a smaller scale, though. Consider a few examples.

HTML

Even hardcore embedded programmers periodically emerge from their command-line sanctuaries to produce documentation, dashboards, and other HTML-based artifacts. For most of us, web applications or at least websites are part of daily work.

Is there any value to a focus on the HTML and its validity, though? If a web app passes all its tests, why check the syntax of the HTML that goes into it?

At least three answers broadly apply in commercial situations:

• HTML validation is generally a fast check. If someone accidentally garbles the punctuation of a tag — “<ui>” for “<ul>” during an edit, say — a validator can typically spot that in under a second, whereas a full test suite typically takes minutes or even hours to complete. The faster your CT finds errors, the quicker your coders are able to turn them around and bring your software back into a “green” state
• It’s widely believed that valid HTML raises search-engine rankings
• Valid HTML is more maintainable

That last point deserves a bit more explanation. It’s easy to come up with fragmentary HTML — source that lacks closing tags, for instance — that meets requirements, in the sense that all major browsers render it as expected. Slightly invalid HTML tends to be fragile, though. Adding another item near an invalidity can result in different browsers producing different results. When HTML is valid, though, and remains valid during maintenance, it’s far more likely that different browsers will continue to render it consistently.

HTML validation’s a good idea, right? In general, yes. Validation helps locate errors quicker and at less expense. I’ve certainly automated HTML validation for quite a few projects.

At the same time, be aware of some of the reasons not to validate your HTML:

• Your HTML is so small or static that it’s not worth the chore of automating its validation
• Your organization doesn’t value validity, and only treats findings as annoyances to ignore
• You have more pressing priorities for testing and choose to focus your attention on domains other than HTML

I most often rely on html5validator, which also knows how to handle CSS and SVG.

Shell

Shell (Bash, PowerShell, and so on) is another rich domain for automation. Even if you think you only have a few dozen lines of shell gluing together the other parts of your project, you should consider putting guards in place for their use:

• In-source configuration of “safe shell” with set -euf -o pipefail and similar
• Static analyzers such as shellcheck
• Study of Bash style guides

These tools are particularly good at spotting unassigned variables, whitespace errors, confused evaluation contexts and subtle spelling errors that can otherwise frustratingly elude capture.

Little pieces of Perl, PHP, VisualBasic, and other now-secondary languages also show up as “glue” in larger projects. Each of these has language-specific checkers that can help ensure the quality of your source.

JSON

JSON is often an alternative to XML. It’s usually simpler and easier to get right. It’s also more likely to be written by an inexperienced hand, and an automated checker quickly finds the out-of-place commas and quotes that plague weakly controlled JSON source.

Markdown

Markdown is roughly comparable to HTML: It’s a format that largely communicates textual content, and like HTML, its consumers are generally forgiving. Markdown renderers tend to show plausible interpretations even with content that isn’t well-formed Markdown.

As with HTML, also, well-styled Markdown is easier to maintain. Markdownlint helps keep Markdown sources properly syntactic.

Also common to both HTML and Markdown is that many projects rely on these languages but maintain templated sources. The individual pages of a website might be written in Jinja, for instance, which an application server then delivers as HTML. It’s often useful to automate separate tests, one to confirm that the template expressions are well-formed, then one to confirm that rendered content is also valid.

And more

As always, part of your opportunity as a tester is to adapt these general descriptions to your own situation. I’ve written special-purpose checkers to confirm stylistic details that the checkers above don’t cover. Also, many other languages — SQL, crontab, C#, and so on — have checkers available. My experience is that disciplined, automated application of these checkers always turns up errors that would have taken far longer to find without them.

The CT you have now is just a starting point. A little energy and creativity applied to your CT can make it considerably more valuable and help you significantly raise the quality of your software products and services.