Why code coverage for Behat?

PHPUnit has built-in several options for generating code coverage data and reports. Behat doesn't. As Konstantin Kudryashov (@everzet) points out in an issue asking for code coverage options in Behat:

Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you're doing code testing with StoryBDD - you're doing it wrong.

He's right I guess. The main issue is that StoryBDD isn't about code, so it doesn't make sense to calculate code coverage for it. Furthermore, the danger of collecting code coverage data and generating coverage reports is that people will start using it as a metric for code quality. And maybe they'll even set management targets based on coverage percentage. Anyway, that's not what this article is about...

In my Advanced Application Testing workshop I just wanted to collect code coverage data to show how different types of tests (system, acceptance, integration and unit) touch different parts of the code. And how this evolves over time, when we're replacing parts of the system under test, or switch test types to achieve shorter feedback loops, etc.

The main issue is that, when it comes to running our code, Behat does two things: it executes code in the same project, and/or (and this complicates the situation a bit) it remotely executes code when it's using Mink to talk to a web application running in another process.

This means that if you want to have Behat coverage, you'll need to do two things:

1. Collect local code coverage data.
2. Instruct the remote web application to collect code coverage data itself, then fetch it.

Behat extensions for code coverage

For the workshop, I created two Behat extensions that do exactly this:

1. LocalCodeCoverageExtension
2. RemoteCodeCoverageExtension

The second one makes use of an adapted version of the LiveCodeCoverage tool I published earlier.

You have to enable these extensions in your behat.yml file:

default:
    extensions:
        Behat\MinkExtension:
            # ...
        BehatLocalCodeCoverage\LocalCodeCoverageExtension:
            target_directory: '%paths.base%/var/coverage'
        BehatRemoteCodeCoverage\RemoteCodeCoverageExtension:
            target_directory: '%paths.base%/var/coverage'
    suites:
        acceptance:
            # ...
            local_coverage_enabled: true
        system:
            mink_session: default
            # ...
            remote_coverage_enabled: true

Local coverage doesn't require any changes to the production code, but remote coverage does: you need to run a tool called RemoteCodeCoverage, and let it wrap your application/kernel in your web application's front controller (e.g. index.php):

use LiveCodeCoverage\RemoteCodeCoverage;

$shutDownCodeCoverage = RemoteCodeCoverage::bootstrap(
    (bool)getenv('CODE_COVERAGE_ENABLED'),
    sys_get_temp_dir(),
    __DIR__ . '/../phpunit.xml.dist'
);

// Run your web application now...

// This will save and store collected coverage data:
$shutDownCodeCoverage();

From now on, a Behat run will generate a coverage file (.cov) in ./var/coverage for every suite that has coverage enabled (the name of the file is the name of the suite).

The arguments passed to RemoteCodeCoverage::bootstrap() allow for some fine-tuning of its behavior:

1. Provide your own logic to determine if code coverage should be enabled in the first place (this example uses an environment variable for that). This is important for security reasons. It helps you make sure that the production server won't expose any collected coverage data.
2. Provide your own directory for storing the coverage data files.
3. Provide the path to your own phpunit.xml(.dist) file. This file is used for its code coverage filter configuration. You can exclude vendor/ code for example.

Combining coverage data from different tools and suites

If you also instruct PHPUnit to generate "PHP" coverage files in the same directory, you will end up with several .cov files for every test run, one for every suite/type of test.

vendor/bin/phpunit --testsuite unit --coverage-php var/coverage/unit.cov
vendor/bin/phpunit --testsuite integration --coverage-php var/coverage/integration.cov
vendor/bin/behat --suite acceptance
vendor/bin/behat --suite system

// these files will be created (and overwritten during subsequent runs):
var/coverage/unit.cov
var/coverage/integration.cov
var/coverage/acceptance.cov
var/coverage/system.cov

Finally, you can merge these files using phpcov:

phpcov merge --html=./var/coverage/html ./var/coverage

You'll get this nice HTML report and for every line it shows you which unit test or feature file "touched" this line:

I hope these extensions prove to be useful; please let me know if they are. For now, the biggest downside is slowness - running code with XDebug enabled is already slow, but collecting code coverage data along the way is even slower. Still, the benefits may be big enough to justify this.