A better PHP testing experience: Introduction

Matthias Noback

This is the introduction to a series of articles related to what I call: the “PHP testing experience”. I must say I’m not really happy with it. And so are many others I think. In the last couple of years I’ve met many developers who experienced a lot of trouble while trying to make testing a serious part of their development workflow. It is, I admit, a hard thing to accomplish. I see many people fail at it. Either the learning curve is too steep for them or they are lacking some insight into the concepts and reasoning behind testing. This has unfortunately led many of them to stop trying.

Even worse, people not only give up writing tests at all, they also tell others about their negative experiences. Those other people have also experienced how hard it is to embrace “development with tests” (not necessarily TDD!). And now they can openly confess that they failed. But they won’t call it a failure of course, they will say that testing isn’t necessary. “I check everything in a browser anyways, so this is only extra work.” Even PHP community leaders like phpclasses.org manager Manuel Lemos (“7 Reasons Why TDD Failed to become Mainstream”)) say things like “if you avoid writing tests for everything from the start, you gain time”. Yes, yes, writing tests for everything takes time, but writing no tests or just a few definitely costs a lot of time too; namely when you need to fix the problems you introduced by not testing in the first place.

If you know me and the way I work, you must know that I really like testing the software I create. Even though I’m an experienced developer, I basically don’t trust myself to do things right and definitely not to keep them right when I’m working on other things that have unanticipated effects on the first things, which suddenly stop being right. So I write tests to be sure that my code does what I expect it to. So for me time nor the amount of energy spent is a reason for not writing tests. For me it is like: if you don’t write tests, how can you even be sure that your software works?

Still, I understand that time can be an issue. I often see myself struggling with the test frameworks I use (PHPUnit and Behat), even after some years of working with them. I think that the testing experience for PHP developers is on average quite bad. The official documentation of test frameworks usually does not a good job at guiding the user in the right direction. The documentation only shows some usage examples, which only demonstrate some cool parts of the syntax it supports, but which are nevertheless far from best practices. Following along with the examples (what else can you do as a developer who has just started to dive into the subject of testing?) will lead to messy test cases which violate coding standards and design principles. And all the time you keep thinking: “Am I doing this right? Am I supposed to put this here? What would Sebastian think of this?” Yet no one answers your questions, you are on your own.

In conclusion: I think there is something wrong about the whole testing business. It seems you can be really good at it, and you can say lots of really smart things about the magical art of testing, but this art of testing seems often to be very far from the actual practices of the average developer. Something needs to be done!

The way I think this can be solved is by making the testing experience a better one. It should be much nicer and easier to work with PHPUnit and Behat than it currently is (by the way, I haven’t used other testing frameworks - except for Lime ;) - so I don’t know if what I’m going to write about in the following articles applies to them too).

There are some things we as a community can do, just to mention a few:

  • We can release code we have written for testing specific things, which are not supported out-of-the-box by the testing framework.
  • We can write about our experiences with testing.
  • We can sit next to our co-worker and show them what we found is a best practice with regard to testing.
  • We can help other developers who are not experienced testers to write their first tests and prevent them from walking down the wrong path.

The PHPUnit testing experience

The following is a quick list of things that bother me with regard to PHPUnit and the way it is often used. I will discuss each of these points more elaborately in the following articles, along with some of the solutions I have in mind.

Unit testing often focuses on assertions

Learning someone to write unit tests is often about learning to use the assertions of a particular test framework. In the case of PHPUnit I’ve heard many people recommend things like “you should only have one assertion per test”, “use the most specific assertion that is available”, “write an assertion first”, etc. I really don’t like all these assertion-centered recommendations. To me tests are not about asserting things, they are about describing behavior in terms of desired results or communication between objects.

Moving the focus away from assertions to descriptions automatically leads to better unit tests. These are some of the things you can do to make your test more descriptive:

  • Use custom assertions, or group assertions.
  • Choose proper names for variables.
  • Use simple mocks, stubs, etc.

Testing interaction with third-party library code

It is often said that “you shouldn’t mock what you don’t own”. This is true, but what I don’t own can often not easily be constructed for use in a test scenario. Still I really need it in the class that is currently my Subject Under Test (SUT). So, mock it anyway? Definitely not, because there is a reason why I shouldn’t mock things I don’t own: I can not be sure that those things work as I think they do. So I have to test part of their behavior in my own test, to ensure that my class and the third-party code work well together. But if I can’t mock the class, then I should introduce an easy way to construct it, without configuring a fancy Dependency Injection Container or adding 100 lines of construction logic to my test class.

Tests of classes that depend on third-party code will be much better maintainable if you:

  • Create a base class with useful helper functions to interact with the third-party code.
  • Or better, don’t use inheritance but composition and introduce a simple dependency injection container.

Tests are supposed to support refactoring of production code but tests themselves need refactoring too

Yes, the ultimate reason for writing tests is that tests enable you to properly and fearlessly refactor your production code. Though it would in theory be possible to only take very small steps while refactoring, and never break the functionality of the code, chances are that you will break it, and it’s possible that you won’t notice it, because you have no test which can verify that it still works as it always did.

But the one thing that many developers forget is that tests themselves need to be refactored too. This is admittedly much more difficult than refactoring production code that is covered by tests: when you modify the structure of a test, you might also accidentally break it, after which it may produce false positives and you would never know that it does (or it might even produce false negatives, in case you will be debugging for quite some time and find nothing, until you turn your eyes to the test itself).

Refactoring tests requires practice and you will soon recognize some patterns in the refactoring process. Sometimes this will even help you to be one step ahead. I will show you some ways in which I’m used to enhance the structure of my tests (which often means just reducing code duplication) by:

  • Using factory methods.
  • Using data providers (and why it is often a good idea to not use data providers).
  • Preparing some things in the setUp() method.
  • Moving technical details out of the way by introducing facade methods.

The Behat testing experience

So far all my concerns have been related to unit testing. But there’s also story, BDD, acceptance, functional, etc. testing which is usually done with the Behat framework. This framework by itself has caused lots and lots of confusion: you could use it to write unit tests, or component tests, even web tests. You can use it to simply replay recorded or written scenario’s. Or to describe the behavior of your application from the point of view of the end users.

Add to that: really bad documentation, also no information at all about the latest stable version (Behat 3). Don’t get me wrong, I really love this tool and I admire its lead developer, but: this really adds up to a steep learning curve and lots of trouble ahead for new users who are just getting started with Behat.

I noticed that Behat tests easily end up:

  • Being fragile.
  • Showing false positives.
  • Being too trivial.

Feature contexts tend to become:

  • Really messy.
  • Lazy (checking only half of the actually desired outcomes or less).

I am not an experienced Behat developer myself (which makes it even more clear which parts of it can pose a problem to new users). I am sure the PHP testing experience with regard to Behat could become much better too. I will try to give some of my personal views and solutions in one of the following articles.