In the previous post we improved the output handling of the test runner, making output optional by using an abstract progress printer. Only our assert_equals function, which by the way should also be moved to the test_framework module, still prints directly to stdout:

function assert_equals(expected, actual, epsilon) result(assertion_failed)
   ! ... 
   assertion_failed = .false.

   if (abs(actual - expected) > epsilon) then
      print *, 'Actual: ', actual
      print *, 'Expected: ', expected
      print *, 'Epsilon: ', epsilon
      print *, 'Reals are not equal'
      assertion_failed = .true.
   end if
end function assert_equals

To fix this, we should upgrade the return type. One idea would be to return a message (variable-length string) instead of a logical, but a more flexible, future-proof alternative is to define a custom type for an error. This would allow us to pass more information back to the caller than just the message. A suitable name would be assertion_failed_t:

In the previous post we’ve successfully split the running of the tests and collecting the test results from handling those results by printing the counters and error stop-ping the test program. We still ask each test procedure to print a small description. We do this to assist the programmer when there is some kind of crash during the test run: they need to be able to find out which test caused it.

Maybe we’ve been writing more “generic” code for our custom test program than expected. We have been gradually, and quite safely, extracting code from a simple “check program”. But now that we’re looking at a pretty generic test runner that we’re going to use for all our future unit tests, we inevitably have to address the concern that we’re writing code that isn’t tested itself. Who tests the test framework? Well, the test framework itself has all the ingredients. However, testing the framework requires that we can run its code in isolation: that we can call one of its functions, and assert that it behaves correctly. Unfortunately, that isn’t possible yet. The code still lives in the main block of the tester program, and it doesn’t allow in-memory inspection of the test results, because it only prints them to the terminal, which we can’t unit-test.

In the previous post, we’ve spent some time improving a number of “temporary” tests. We introduced an assertion function to compare two real values, and we prevent failing tests from stopping the entire test program.

Moving tests to their own procedures

The approach of adding tests directly in the main test program block doesn’t really scale, as the saying goes. polyline_length may be a simple function, but for more complicated functions with multiple branches and loops, we’d have to write many of these tests. The test program keeps growing, and eventually it becomes a mess. It doesn’t help that all local variables have to be declared at the top. Even if that wasn’t needed, it isn’t very clear where a test starts or ends. Everything happens in the same scope, potentially becoming a memory management issue too. This also makes it hard to delete tests we no longer want. Removing lines from a big test program likely breaks other tests, or we forget to remove things we no longer need.

How can we know that the function we wrote, works as intended? We could run it, and manually verify its correctness. The simplest way to do this is to call the function in the main program block, print the output, and compare it with what we expect. Say our function calculates the length of a polyline, stored as a two-dimensional array of reals, representing (x,y) coordinates:

pure function polyline_length(coordinates) result(length)
   real(kind=real64), dimension(:, :), intent(in) :: coordinates

   real(kind=real64) :: length

   real(kind=real64) :: distance
   integer :: index

   length = 0.0_real64

   do index = 1, size(coordinates, 1) - 1
      distance = sqrt((coordinates(index, 1) - &
                       coordinates(index + 1, 1))**2 + &
                      (coordinates(index, 2) - &
                       coordinates(index + 1, 2))**2)
      length = length + distance
   end do
end function polyline_length

Temporary test programs

We could modify the main program block of our actual program, but it’s a lot simpler and safer to create a separate “throw-away” test program, with only the code we need. We’d write a short test program that sets up some coordinates, calls the function, then prints the result:

Unbalanced test suites

Having spoken to many developers and development teams so far, I’ve recognized several patterns when it comes to software testing. For example:

1. When the developers use a framework that encourages or sometimes even forces them to marry their code to the framework code, they only write functional tests - with a real database, making (pseudo) HTTP requests, etc. using the test tools bundled with the framework (if you’re lucky, anyway). For these teams it’s often too hard to write proper unit tests. It takes too much time, or is too difficult to set up.
2. When the developers use a framework that enables or encourages them to write code that is decoupled from the framework, they have all these nice, often pretty abstract, units of code. Those units are easy to test. What often happens is that these teams end up writing only unit tests, and don’t supply any tests or “executable specifications” proving the correctness of the behavior of the application at large.
3. Almost nobody writes proper acceptance tests. That is, most tests that use a BDD framework, are still solely focusing on the technical aspects (verifying URLs, HTML elements, rows in a database, etc.), while to be truly beneficial they should be concerned about application behavior, defined in a ubiquitous language that is used by both the technical staff and the project’s stakeholders.

Please note that these are just some very rough conclusions, there’s nothing scientific about them. It would be interesting to do some actual research though. And probably someone has already done this.

Fernando Arconada interviewed me about the subject of testing. He is writing a book about testing Symfony2 applications: Testing para Aplicaciones Symfony2. Fernando will translate this interview to Spanish and and add it to his book, together with the articles in my A better PHP testing experience series.

Who is Matthias Noback?

I’m a PHP developer, writer and speaker. I live in Zeist, The Netherlands, with my girlfriend, a son of 9 and our newborn daughter. Currently I have my own business, called Noback’s Office. This really gives me a lot of freedom: I work as a developer on one project for about half of the week and in the remaining time I can either spend some time with my family or write blog posts, or finish my second book.

Apparently not everyone agrees on how to unit test their Symfony2 controllers. Some treat controller code as the application’s “glue”: a controller does the real job of transforming a request to a response. Thus it should be tested by making a request and check the received response for the right contents. Others treat controller code just like any other code - which means that every path the interpreter may take, should be tested.