Review

After reading "Discovery - Explore behaviour using examples" by Gáspár Nagy and Seb Rose, I picked up another book, which I bought a long time ago: "Fifty Quick Ideas to Improve Your Tests" by Gojko Adzic, David Evans, Tom Roden and Nikola Korac. Like with so many books, I find there's often a "right" time for them. When I tried to read this book for the first time, I was totally not interested and soon stopped trying to read it. But ever since Julien Janvier asked me if I knew any good resources on how to write good acceptance test scenarios, I kept looking around for more valuable pointers, and so I revisited this book too. After all, one of the author's of this book - Gojko Adzic - also wrote "Bridging the communication gap - Specification by example and agile acceptance testing", which made a lasting impression on me. If I remember correctly, the latter doesn't have too much practical advice on writing goods tests (or scenarios), and it was my hope that "Fifty quick ideas" would.

First, a few comments on the book, before I'll highlight some parts. I thought it was quite an interesting book, covering several underrepresented areas of testing (including finding out what to test, and how to write good scenarios). The book has relevant suggestions for acceptance testing which are equally applicable to unit testing. I find this quite surprising, since testing books in general offer only suggestions for a specific type of test, leaving a developer/reader (including myself) with the idea that acceptance testing is very different from unit testing, and that it requires both a different approach and a different testing tool. This isn't true at all, and I like how the authors make a point of not making a distinction in this book.

The need to describe why

As a test author you may often feel like you're in a hurry. You've written some code, and now you need to quickly verify that the code you've written actually works. So you create a test file which exercises the production code. Green light proves that everything is okay. Most likely you've tested some methods, verified return values or calls to collaborating objects. Writing tests like that verifies that your code is executable, that it does what you expect from it, but it doesn't describe why the outcomes are the way they are.

In line with our habit of testing things by verifying outputs based on inputs, the PhpStorm IDE offers to generate test methods for selected methods of the Subject-Under-Test (SUT).

I always cry a bit when I see this, because it implies two things:

1. I've determined the API (and probably wrote most of the code already) before worrying about testing.
2. I'm supposed to test my code method-by-method.
3. A generated name following the template test{NameOfMethodOnSUT}() is fine.

At the risk of shaming you into adopting a test-first approach, please consider how you'll test the code before writing it. Also, aim for the lowest possible number of methods on a class. Make sure the remaining methods are conceptually related. The SOLID principles will give you plenty of advice on this topic.

Anyway, if we're talking about writing tests using some XUnit framework (e.g. JUnit, PHPUnit), don't auto-generate test methods. Instead:

Name the test class {NameOfConcept}Test, and add public methods which, combined, completely describe or specify the concept or thing. I find it very helpful to start these method names with "it", to refer to the thing I'm testing. But this is by no means a strict rule. What's important is that you'll end up with methods you can read out loud to anyone interested (not just other developers, but product owners, domain experts, etc.). You can easily check if you've been successful at it by running PHPUnit with the --testdox flag, which produces this human-readable description of your unit.

Rules and examples

There's much more to it than following this simple recipe though. As the book proposes, every scenario (or unit test case), should first describe some rule, or acceptance criterion. The steps of the test itself should then provide a clear example of the consequences of this rule.

Rules are generic, abstract. The examples are specific, concrete. For instance, a rule could be: "You can't schedule a meeting with someone who already has a meeting at the same time." An example of this rule could be: "Given Judy attends a meeting that's scheduled from 2PM until 4PM, when Mike schedules a 1-hour meeting at 3PM, then he won't be able to invite Judy for it." Rules are often known as "acceptance criteria". Hence, if you write a scenario for an acceptance test, you start the scenario by describing the rule, and you then show an example that demonstrates the rule in the context of the application you're building, e.g.

Scenario: You can't schedule a meeting with someone who already has a meeting at the same time.

  Given Judy attends a meeting that's scheduled from 2PM until 4PM
   When Mike schedules a 1-hour meeting at 3PM
   Then he won't be able to invite Judy for it

I find that very often the original developers of the code know the rule that's involved in a scenario, but don't write it down. They assume the reader will be able to extract the rule from the example by reading the scenario. This may work for some obvious scenarios, but in other cases it won't. It may be obvious to you now, but it won't be to someone else next year. So, as a developer, make sure to write and automate scenarios like the one above, but also document what you've learned about your application and its domain, by writing down all the rules that are involved.

The opposite problem may also occur by the way. It happens when the tests only specify the rules, and the examples just repeat the rules. For instance:

Scenario: You can't schedule a meeting with someone who already has a meeting at the same time.

  Given someone attends a meeting
   When someone else schedules a meeting at the same time
   Then they can't invite them to this meeting

Besides the fact that the automation code behind these steps will be horrible,

Unless examples are making things more concrete, they will mislead people into a false assumption of completeness, so they need to be restated.

Decoupling how something will be tested from what is being tested

A big issue with most "acceptance test suites" I've seen, is that they mix what will be tested with how it's tested. If you want to separate these two things in software, the answer is usually: abstraction. And it's missing if you're using built-in step definitions for navigating a website by clicking links, looking for HTML elements and analyzing their contents. The biggest issue is that these scenarios don't belong in acceptance tests. There's nothing about acceptance criteria in there. These scenarios are more like exploratory tests, the actions of which have been recorded and automated.

From the perspective of maximizing the quality of your tests, the issue isn't that running end-to-end tests exercising the actual UI of the application is slow and fragile. Even though this is a bad thing for the longer term, the issue today is that these automated scenarios break for many reasons that are unrelated to the acceptance criteria themselves. The need to update the scenario when a URL, the structure of the HTML of a page changes (or even when there's a change in the CSS or JavaScript that's used), is annoying. It slows developers down, and soon they will feel that the tests don't add much value, because they break more often than they are helpful.

Decoupling how something will be tested from what is being tested significantly reduces future test maintenance costs.

The true power of scenarios comes from being able to describe behavior in a technology-agnostic way. This allows us to work together with stakeholders and domain experts on coming up with good examples for the rules they apply, and the acceptance criteria they have defined for the application. It helps defining counter examples, and refining those rules. I've found that it even helps showing the incorrectness of certain rules, or rephrasing them using more appropriate terms.

How to write good scenarios? Given-When-Then

Writing good scenarios requires good writing. Unfortunately, writing itself often invokes a sense of self-awareness, which easily leads to getting stuck. I'm getting stuck writing this post as we speak, so there you go. Anyway, the general idea of writing a scenario is that you follow the template Given-When-Then. This should lead you in the right direction. However, it often leads people astray. You may end up asking yourself if a certain step is part of 'Given' or 'When'. You may write just one scenario which shows every aspect of a feature, and wonder if you need to write any more scenarios. You may be writing lots and lots of scenarios, each copy-pasted from the one before it, with just some minor variations. Anyway, it's clear that we all need some proper advice about writing those scenarios!

I think the following quotes from the book give some really helpful advice. The first one is about grammar:

A good trick that prevents most accidental misuses of Given-When-Then is to use past tense for 'Given' clauses, present tense for 'When' and future tense for 'Then'. This makes it clear that 'Given' statements are preconditions and parameters, and that 'Then' statements are postcondition and expections.

Besides using the present tense only for the 'When' clause, also limit the use of the active tense to the 'When' clause.

Make 'Given' and 'Then' passive - they should describe values rather than actions. Make sure 'When' is active - it should descibe the action under test.

You can have several 'Given's, or 'Then's. Keep them small and compact though, and prevent long lists of steps. Having many 'Given' steps will confuse the reader, who will wonder: which preconditions are essential? Are they somehow related? Having many 'Then' steps likely means that the scenario tries to test multiple things in one go. This requires a rewrite by splitting the scenario into two or more scenarios, each with a clear purpose. About that:

Each scenario should ideally only have one 'When' clause that clearly points to the purpose of the test.

Conclusion

There's more to write about the advice given by this nice "Fifty quick ideas" book. I'm saving that for another blog post.