Should We Automate All of Our Feature Scenarios?

Behavior-driven development (BDD) is a methodology in which we specify an application by first describing its behavior in the form of human-readable examples. Here’s a simple example that describes how one element of ATM withdrawals should work:

Behavior description of ATM withdrawal

Feature:
ATM actions

Background:
The account holder has successfully selected an account using their debit card

Scenario:
Rejects attempt to withdraw more funds than available in the account holder’s account

• Given the account holder has an account balance of $100
• When the account holder attempts to withdraw $110
• Then the ATM displays the message “Insufficient funds; transaction cancelled”
• And the ATM ejects the card

If you’ve built software long enough, you might note some similarities to the concept of using use cases to describe the requirements of a system. So what’s different? On the surface, not much. The terminology (features and scenarios) is a little different, but a scenario in BDD is much like the narrative of a use case: An actor (in the ATM example, the account holder) interacts with the system, and the system responds in some accordant fashion. The use case goal (its title) is similar to the scenario summary description—both represent the end state that should occur when an actor interacts with the system.

Use cases were intended to be a different way of driving the design and implementation of a system from its requirements. The set of use cases represents the comprehensive set of end goals that all possible actors can accomplish with a system. The narrative form provides the best way to capture and describe the steps (requirements) needed to accomplish these end goals. Wikipedia tells us that the “main success scenario of each use case provides everyone involved with an agreement as to what the system will basically do and what it will not do.”

The key word in the Wikipedia description of use cases is agreement. In order to deliver successful software, we—product owners, testers, and developers—must all be on the same page regarding how it should behave. Use cases give us a focal point for agreement. We can quibble about the narratives and pin down a new one: “No, the ATM should not eject the card; it should instead prompt the user for another action.”
Still, there are a couple of key differences between use cases and the examples of BDD:

• Organizationally, BDD scenarios don’t match up one to one with use cases; a discrete scenario might be an alternate case for a single use case
• Use cases aren’t specific examples of one user’s interactions; they attempt to generalize the behavior.

For our ATM example, a use case might look like this:

Use case of ATM withdrawal

Actor:
Account holder

Goal:
Withdraw money from account

Narrative:
(Happy path withdrawal case here)

Alternate path:

• The account holder attempts to withdraw more funds than available in the account
• The system responds with an error message and ejects the card

Note that the alternate path narrative for this use case doesn’t mention specifics (e.g., an account balance of $100).

Where Did the Use Cases Go?

I rarely encounter teams incorporating use cases into their process anymore, so something must not be so great about them. What’s not to like?

Well, for one, teams often got into what might be called “analysis paralysis.” They debated the use cases endlessly, sometimes in an obsessive-compulsive manner. I remember at least a few debates over the granularity of the use cases, and worse, sometimes we would argue incessantly over wording: “We should use the word will instead of should here.”

The bigger problem is that the use cases are often misinterpreted. No matter how much we wordsmith, it’s possible to misunderstand the English language. Take this classic programmer joke:

A programmer is going to the grocery store. Her spouse says, “Buy a gallon of milk, and if there are eggs, buy a dozen.” On arriving back home after the grocery trip, the spouse angrily asks, “Why did you get 12 gallons of milk?” The programmer replies, “They had eggs.”

One customer described to me their painful experience with use cases. The product owner had sent some use cases to an offshore team for development; what came back a couple of weeks later wasn’t quite what they expected. Through a series of updates to the use cases, the development team kept trying to build what they thought the product owner was asking for. Unfortunately, this back-and-forth took three more two-week iterations before the developers got it right.

Resorting to “legalese” provides one way to remove ambiguity, but we then often produce inscrutable prose that no one can understand. Using examples in BDD can begin to minimize opportunities for misinterpretation. Here’s a set of examples in table form that represent the intended outcome of the grocery trip:

Has eggs?	Purchase
No	1 gallon milk, no eggs
Yes	1 gallon milk, 12 eggs

Granted, the example is a little silly, but distilling the ambiguous prose to a set of expected outcomes for a number of scenarios often provides immediate clarity. As a member of the development team, if you’re still uncertain about the appropriate behavior, you can always request more scenarios from the product owner.

One Step Beyond

Most teams adopting BDD use a tool such as Cucumber or FitNesse to support automating these examples. The ATM example from earlier is written to support the use of Cucumber or SpecFlow, an open source port of Cucumber for .Net. As such, each step of its scenario is automated to support an executable test.

Behind the scenes, each step description (for example, “When the account holder attempts to withdraw $110”) gets translated into code that drives the system you’re testing. Steps that verify things (indicated by statements starting with “Then,” such as “Then the ATM displays the message”) get translated into code that asks the system a question, then compares the answer with the expected outcome. If the answer matches the expectation, the scenario passes; otherwise, it fails.

Automating BDD scenarios allows a development team to demonstrate that they’ve met product owner expectations. We can execute these examples at the click of a button. If the scenario examples all pass, the development team knows their work is complete; if any one fails, the development team knows they have more work to do. The rapid feedback provided by running these automated examples supports our interest in short-cycle iterative development: After adding a new feature, we can gain the confidence to ship it by making sure all the examples execute successfully.

Oh, No—We Can’t Automate This!

Automating the examples makes continuous delivery feasible. But the most important value we get from crafting the scenarios in BDD is their ability to act as a focal point for understanding and negotiation. This conforms to our Wikipedia description for use cases as “an agreement as to what the system will basically do and what it will not do.”

Sometimes it might not even be possible to automate any scenarios. Many years ago, I worked to help a customer introduce BDD into their process for a production web-based system that they were continually updating. I sat and worked through a few feature examples with the business analysts to help them understand what they should be doing. The customer went off on their own and proceeded to flesh out numerous additional scenarios. Meanwhile, I attempted to automate the first scenario.

Unfortunately, my customer’s system wasn’t architected with testing in mind. I made a number of attempts to find a clean entry point that would allow us to automate the examples. But there were no clean APIs to hook into, and the web front end had some complexities that made it impossible to easily “robot-drive” the system through its user interface.

I felt awful about having to tell the BAs that we couldn’t automate anything. We’d already spent many hours introducing BDD, and I had to tell them that there wouldn’t be any cost-effective way to automate the numerous scenarios they’d created.

I was surprised when they told me not to worry. They were still excited about the BDD techniques they’d learned. The most useful proposition for them was a way to consistently approach pinning down, negotiating and organizing the requirements. They indicated that the automation would be nice, but for the time being, they found just the collaborative part of BDD to be its most useful element.

In BDD, automation is not the primary goal. Yes, automation may be essential if you want to do continuous delivery, and yes, automation is where you see great success with BDD. But the first goal is to understand how to use BDD as a means of getting on the same page. And if you can’t get that far, automation isn’t going to help you much anyway.