Several articles later, it turns out the little test framework has grown very fast. We’re looking at around 400 lines of code in a single test_framework module. In my opinion, this requires too much scrolling and jumping through the code to understand what’s going on or make changes. I’m aware “real-world” projects suffer from files that are a lot larger, but they are in a lot of trouble because of it. It’s smart to split large modules into smaller ones at a much earlier stage. As we discussed before, the following guidelines may be used when doing so:

The right approach to software design, in my opinion, is to work with what you have, build more functionality on top of it, then realizing there are design issues, then fixing those issues by redesigning ad hoc. Where, in my experience, most software design efforts go wrong is:

• We realize there are design issues, yet we don’t fix them. Maybe it’s too scary.
• We fix design issues, but too early, when we can’t yet know if the new design is better. We lack feedback from actual use.

In this article series, I’m happy to report, I haven’t spent too much time designing upfront. But now I want to tackle some issues, that I encountered while adding more tests. Let’s look at one of our tests:

When expanding our test suite, we’ll certainly encounter the need for an assert_equals function that works with other types than real(kind=real64). We’d want to compare values of type logical, character, integer, etc. For example, say we have a utility function str_to_upper for which we are adding a unit-test in the test_string module. This function is supposed to convert the letters in a string to their uppercase alternative. The test looks like this:

In the previous post we improved the output handling of the test runner, making output optional by using an abstract progress printer. Only our assert_equals function, which by the way should also be moved to the test_framework module, still prints directly to stdout:

function assert_equals(expected, actual, epsilon) result(assertion_failed)
   ! ... 
   assertion_failed = .false.

   if (abs(actual - expected) > epsilon) then
      print *, 'Actual: ', actual
      print *, 'Expected: ', expected
      print *, 'Epsilon: ', epsilon
      print *, 'Reals are not equal'
      assertion_failed = .true.
   end if
end function assert_equals

To fix this, we should upgrade the return type. One idea would be to return a message (variable-length string) instead of a logical, but a more flexible, future-proof alternative is to define a custom type for an error. This would allow us to pass more information back to the caller than just the message. A suitable name would be assertion_failed_t:

In the previous post we’ve successfully split the running of the tests and collecting the test results from handling those results by printing the counters and error stop-ping the test program. We still ask each test procedure to print a small description. We do this to assist the programmer when there is some kind of crash during the test run: they need to be able to find out which test caused it.

Maybe we’ve been writing more “generic” code for our custom test program than expected. We have been gradually, and quite safely, extracting code from a simple “check program”. But now that we’re looking at a pretty generic test runner that we’re going to use for all our future unit tests, we inevitably have to address the concern that we’re writing code that isn’t tested itself. Who tests the test framework? Well, the test framework itself has all the ingredients. However, testing the framework requires that we can run its code in isolation: that we can call one of its functions, and assert that it behaves correctly. Unfortunately, that isn’t possible yet. The code still lives in the main block of the tester program, and it doesn’t allow in-memory inspection of the test results, because it only prints them to the terminal, which we can’t unit-test.

In the previous post, we’ve spent some time improving a number of “temporary” tests. We introduced an assertion function to compare two real values, and we prevent failing tests from stopping the entire test program.

Moving tests to their own procedures

The approach of adding tests directly in the main test program block doesn’t really scale, as the saying goes. polyline_length may be a simple function, but for more complicated functions with multiple branches and loops, we’d have to write many of these tests. The test program keeps growing, and eventually it becomes a mess. It doesn’t help that all local variables have to be declared at the top. Even if that wasn’t needed, it isn’t very clear where a test starts or ends. Everything happens in the same scope, potentially becoming a memory management issue too. This also makes it hard to delete tests we no longer want. Removing lines from a big test program likely breaks other tests, or we forget to remove things we no longer need.

How can we know that the function we wrote, works as intended? We could run it, and manually verify its correctness. The simplest way to do this is to call the function in the main program block, print the output, and compare it with what we expect. Say our function calculates the length of a polyline, stored as a two-dimensional array of reals, representing (x,y) coordinates:

pure function polyline_length(coordinates) result(length)
   real(kind=real64), dimension(:, :), intent(in) :: coordinates

   real(kind=real64) :: length

   real(kind=real64) :: distance
   integer :: index

   length = 0.0_real64

   do index = 1, size(coordinates, 1) - 1
      distance = sqrt((coordinates(index, 1) - &
                       coordinates(index + 1, 1))**2 + &
                      (coordinates(index, 2) - &
                       coordinates(index + 1, 2))**2)
      length = length + distance
   end do
end function polyline_length

Temporary test programs

We could modify the main program block of our actual program, but it’s a lot simpler and safer to create a separate “throw-away” test program, with only the code we need. We’d write a short test program that sets up some coordinates, calls the function, then prints the result:

We’ve encountered several ways of designing functions in a way that allows them to fail for some reason, without stopping the program, or making it otherwise risky or awkward to use the function. We introduced the error_t type which is very flexible. It can be used to provide some information to the caller, helping them understand what went wrong and how it can be fixed. By allowing errors to be wrapped inside others, we can create chains of errors that describe the problem at various abstraction levels. It gives back control to the user: how do they want to deal with an error? Would they like to try something else? Or, in the end, should we just stop trying and quit te program?

Parsing an array of strings to a polyline should fail if one of the strings could not be parsed to a string. But it should also fail if the resulting array of points is empty. Or at least, if that makes sense in the application’s domain. You could say that a polyline with no points is still a polyline, just like an empty set is still a set. Similarly, in some scenarios it may be okay for a polyline to have just a single point. For now, let’s skip the mathematical discussion and take this as a practical rule that we want to enforce: a polyline has at least 2 points. The type we currently have can’t give us such a guarantee:

Parsing a string to a point may not be successful, and we were able to communicate that to users of parse_point() by returning a custom type called point_or_error_t.

What if we want to parse not just one point but a set of points, forming a polyline, from a list of strings? The result of parsing each string separately is a point or an error. Only if the list of those point_or_error_t values contains no error at all should we consider turning the points into a polyline. If any of the values is an error, we shouldn’t make a polyline of the other valid points, but instead return an error: “Could not create polyline from strings. Previous error: Could not extract two decimal numbers from …” With this error we communicate what we were trying to do at the highest conceptual level (create a polyline), and what lower-level error caused this to fail.

We looked at optionally returning a value from a function. In the case of average it was an average value, or no value at all. In some more elaborate functions, the “no result” case needs some more explanation to the caller of the function. We may feel the need to communicate what went wrong, or for what reason. This gives the user the opportunity to pass better input next time, or fix some external issue that the program can’t fix itself (e.g. a file not being readable).

In the previous post we introduced an optional_real_t type to be the return value of an average function, because depending on the size of the array of numbers passed to it this function may or may not return a real value.

What if we could just prevent people from calling the average function when they don’t have any numbers? Maybe we’re lagging too far behind if we have to verify the size of the array inside the average function.

In the previous post we’ve looked at several ways to communicate to the user of a function that something could go wrong, or that maybe the function couldn’t produce the result that the user was looking for. In the case of an average function, we’d want to communicate to the user that in some cases the function can’t produce a meaningful result, namely if the array of numbers is empty. This isn’t really an error, it’s just that we don’t have a way to answer the question “what’s the average of this array?”. To communicate that with the function signature, we tried adding a “success” return value, making the calculated average an intent(out) argument. The function modifies the variable passed by the user to contain the average value, but only if the array is not empty:

Fortran doesn’t have exceptions. You can’t jump out of a function when something unexpected happens, then reflect on what happened outside the function, with the help of a try/catch block. As an alternative, intrinsic (built-in) procedures and constructs like write, read and open allow you to pass an integer variable as one of its arguments, to which it will assign a specific value indicating whether an error occurred. For example, when you try to open a file that doesn’t exist, you will get a non-zero value in the variable passed as the argument for iostat:

We’ve implemented filter and map operations. The results of both these operations are new arrays. Yet another common operation on arrays is the so-called reduction operation; we “reduce” multiple values to a single value. The most obvious example is calculating the sum of an array of integers: the input is an array (or list) and the output is a single integer:

pure function sum_all(numbers) result(res)
   integer, dimension(:), intent(in) :: numbers
   integer :: res

   integer :: i

   res = 0

   do i = 1, size(numbers)
      res = res + numbers(i)
   end do
end function sum_all

Just like with filter and map we can imagine a generalization of this process, which involves looping over the array elements and building up a single return value along the way. We start with the existing sum_all function. The first step is to let the user decide what the return value should be, which is the logic that happens inside the do loop. We extract a function for this:

In the previous post we looked at this dream scenario:

list([1, 3, 5, 7]) % filter(divisible_by(3)) % map(square)

As mentioned, we can’t chain function calls like this. But we can put intermediate results in local variables and achieve something similar:

integers = list([1, 3, 5, 7])
divisible_by_3 = integers%filter(divisible_by(3))
squared = divisible_by_3%map(square)

If the intermediate values are all the same type, we can reuse the variable:

integers = list([1, 3, 5, 7])
integers = integers%filter(divisible_by(3))
integers = divisible_by_3%map(square)

To enable this syntax, it’s clear that we need to introduce a derived type for a list of integers that has type-bound procedures filter and map. These functions should each return a new list, so we can call any of these functions again on the new list. Let’s start with an integer_list_t which contains an array of integers. We also provide a generic list interface with a specific procedure create_integer_list for creating the integer_list_t:

We considered the problem of filtering elements in an array, keeping only the ones we want. A decision about that is made by a function we pass as an argument.

A similar problem, but with its own challenges, is when we have an array of values and want to create a new array, with an equal number of elements, but each element has been transformed in some way. For instance, say we have an array of integers and want to square each integer. A procedural approach looks as follows:

We have a reusable filter function to which we pass the name of a procedure that matches a given interface. But can we also do something like this?

filter([1, 2, 3, 4], is_divisible_by(3))

Closures

This is a very common thing in functional programming. It involves something called higher-order functions. This means that a function may itself return a function. In this case, is_divisible_by returns a function that can be passed to filter. This returned function also needs to remember the value 3 that was passed to it. When it’s executed for each value in the integer array, it will check if that value is divisible by 3. In other languages such a function is called a closure; it’s a function that can be passed around as a value. Besides the function itself, it also has the data from the context where it was created - “bound” to it. In “ideal Fortran”, we would write something like this:

We have a working filter function for integers. Now can we generalize it? It would be nice if we could filter other types of arrays, like real arrays.

Generic filtering

As mentioned before, Fortran doesn’t have support for generics, which would allow you to use some kind of “placeholder” for types (e.g. [value_type]):

pure function filter(values, filter_func) result(res)
   [value_type], dimension(:), intent(in) :: values

   interface
      pure function filter_func(value) result(keep)
         implicit none(type, external)

         [value_type], intent(in) :: value
         logical :: keep
      end function filter_func
   end interface

   [value_type], dimension(:), allocatable :: res

   integer :: i

   res = pack(values, [(filter_func(values(i)), i=1, size(values))])
end function filter

You could then use filter on an array with any type of value, and the filter_func’s value argument would have to be of that same type, and the return type would be of that type too.

Once introduced to Fortran, we are left wondering what kind of language it really is. Traditionally, it’s been used as a procedural language, or a “Do this, do that” language as I’d like to call it. You can do some Object-Oriented programming with it, but it’s somewhat clunky, verbose, and not very “idiomatic”. (That’s not to say that you shouldn’t do it!) The same goes for Functional programming. If we’d like to write Fortran code that follows this paradigm, it becomes even more awkward. You’ll need several advanced techniques to make it work, and then there are still many limitations. The inability to chain function calls, the lack of tail call optimization and generic type-support, and the obstacles you have to conquer to get some kind of delayed execution; all of these sound like good reasons for not-even-trying…

In the previous post we properly encapsulated the integer level with a log_level_t derived type, achieving type safety.

Now, let’s consider the following feature request: we want to show the log level in front of the message. We could do it with decoration of course, but in the scope of this article, let’s do it in the log() subroutine directly. In this case we’re “lucky” that we can access the module-private data component level, so we can do a select case on it to determine what string should be printed:

In the previous post we introduced a derived type log_level_t. The type of data passed to the log() procedure is still an integer though:

subroutine log(message, level)
   character(len=*), intent(in) :: message
   integer, intent(in) :: level

   ! ...
end subroutine log

This doesn’t stop anyone from passing a completely meaningless integer value as the level argument to log().

Increasing type-safety

We’d like to increase type-safety and the way to do it is by using a more specific type. In our case, it would be great if we could use log_level_t as the argument type for level:

In the post about decoration we declared log levels as follows:

module logging_base
   ! ...
   integer, parameter, public :: LOG_DEBUG = 0
   integer, parameter, public :: LOG_INFO = 1
   integer, parameter, public :: LOG_WARN = 2
   integer, parameter, public :: LOG_ERROR = 3
   integer, parameter, public :: LOG_FATAL = 4
   ! ...
end module logging_base

This is somewhat useful, but requires a dedicated only mention when importing each of these constants, which isn’t very nice:

program main
   use logging_base, only: log, LOG_DEBUG, LOG_WARN
    
   ! ...
   call log('A debug message', LOG_DEBUG)
    
   ! ...
   call log('A warning', LOG_WARN)
end program main

Although we could live with that, the bigger issue remains: this is not a type-safe solution. Any integer may be passed as an argument for level, and the compiler wouldn’t warn us. For example:

Fortran projects are famous for their large modules. Actually, we may also find very large files in legacy projects written in other languages, like Java, PHP, etc. These projects sometimes have files with thousands of lines of code. Fortran projects suffer more from large files I’d say, because:

• IDE support is not great. It’s often still hard to navigate to definitions or usages. There isn’t any support for automated refactorings, like extract function, move function, add parameter, etc. You often can’t get a clear overview of the structure/outline of a file.
• There are no strict indentation rules, making it easy to create a complete mess. Not everyone uses code formatting tools. Code formatting tools that exist and are used, aren’t always very good or easily configurable.
• Fortran code can be hard to read, partially because a lack of curly braces. It’s hard to visually recognize blocks of code. Another reason is that commonly there’s no distinction in letter casing between type names, function names, variable names, etc. Other languages may use CamelCase for types, CAPS for constants, pascalCase for methods, snake_case for functions, etc. In Fortran code, it’s all snake_case.

If there’s anything we can do to make Fortran code easier to deal with, it’s to split the code into smaller parts. We need to have many more modules, each with distinct topics and concerns, so we can get a quick overview of the application’s domain and capabilities. Additionally, this will help us find the right place when we want to make changes to the code.

In the previous post we saw how to use an abstraction to compose an aggregation of services of that same abstraction. There we simply delegated a call, adding no specific behavior. But we might as well add some new behavior before or after delegating the call.

As an example, let’s try to implement a new feature in the logging module: we want to add a timestamp in front of each message, so the output becomes something like this:

With our new service abstraction logger_t we are able to easily implement more features for our logger. We can do this without affecting current user code, because their code relies only on the abstraction, never on concrete implementations.

A terminal output logger

One thing we can do is define a second logger type, one that prints each message to the terminal output (stdout). Being a logger, it should also extend the abstract logger_t type, and provide an implementation for log():

With the new file_logger_t type we were able to model a logger service as a derived type. Now it’s time to make the service abstract. That is, to leave out the implementation detail that it’s a file logger (not a terminal output logger, etc.). Users should be able to use this abstract service to log any message, to whatever the application has been configured to log to.

An abstract derived type

Allowing a user to depend on an abstract service requires the use of an abstract type. The user can import this abstract type in their module or program. Meanwhile, the factory function for the abstract logger will actually return the concrete file logger, which extends the abstract logger. For object-oriented programmers this may be a familiar approach, but it’s not a very common Fortran practice. In code it looks as follows:

We’ve seen how to define a derived type to represent a point in 2D space. We were able to add a type-bound procedure to it, and finally to make its components private and define a custom constructor for it.

In essence there are two categories of derived types:

1. Derived types that hold some data. Their type-bound procedures are solely dealing with this data, and may produce valuable information with it. The point is an example of such a derived type: we can use its data to make geometrical calculations, or we can combine it together with other points into lines, polygons, etc. This all happens in memory. In typical business applications you would consider entities, value objects and data transfer objects (DTOs) to be in this first category.
2. Derived types that can perform some task that deals with things external to the application, like a file, the terminal, a network connection, etc. Such a type doesn’t hold data inside. It is able to fetch data, or send/store data. In business applications such a type is often called a service.

In object-oriented programming, service types usually offer an abstraction. This allows the client to decouple from its specific implementation details. For example, take a logger service; it’s often beneficial for the user that they don’t have to be concerned about where the logs are written to (file, screen, log aggregation server, etc.). The user only wants to call a simple log() function to log something to whatever has been configured at some higher level. Introducing a logger abstraction will take a few posts from here, but it’s good to know that service abstraction is the goal.

In the previous post we have worked on the point_t derived type, turning the module procedure distance into a type-bound procedure:

module geometry
   ! ...

   type :: point_t
      real :: x
      real :: y
   contains
      procedure :: distance => point_distance
   end type point_t

contains

   pure function point_distance(point_1, point_2) result(the_distance)
      class(point_t), intent(in) :: point_1
      
      ! ...
   end function point_distance

end module geometry

Private data components

Type-bound procedures can help us tie relevant behaviors (procedures) to derived types. It also allows us to let the derived type keep its data to itself. This is often called data hiding, or encapsulation of state. Nothing outside the module where the derived type is defined will have (read or write) access to its data components. This can be accomplished by adding the attribute private to each data component:

In the previous post we defined a derived type point_t with real data components x and y, to represent a point in 2D space. We also wrote a function to calculate the distance between two such points:

module geometry
   implicit none(type, external)

   private
   public :: point_t
   public :: distance

   type :: point_t
      real :: x
      real :: y
   end type point_t

contains

   pure function distance(point_1, point_2) result(the_distance)
      type(point_t), intent(in) :: point_1
      type(point_t), intent(in) :: point_2
      real :: the_distance

      the_distance = sqrt((point_2%x - point_1%x)**2 + &
                          (point_2%y - point_1%y)**2)
   end function distance

end module geometry

This allowed us to pass point_t instances as function arguments to distance, in another module or in the program:

We’ve seen types and variables and functions and subroutines. Now we can get to the next level and combine both variables and procedures in a so-called derived type.

Declaring a derived type and its data components

Object-oriented programming languages have classes. It took me some time to understand that classes can be seen as “extensions of the type system”. There are often primitive types like strings, integers, etc. and we can group them as properties of a class. Thereby, the class becomes some kind of advanced, composite type.

So far we’ve put code inside the main program block of our application. Soon the need arises to move blocks of code to a better place: a procedure with a proper name living inside a module which groups related procedures by topic.

Fortran has two types of procedures:

• Functions, with arguments and a return value
• Subroutines, which are functions without a return value

Subroutines

Let’s define a new subroutine in a module:

We’ve looked at the program and module blocks that can hold our code. Now we’ll find out where to put our data.

Declaration comes first

In Fortran, every time we want to store some value in a variable, we have to explicitly declare the type and the name of the variable first. This has to happen before any other executable statement, but after imports and other declarations like the implicit none statement:

We have a basic Fortran application and a working environment to edit, compile and run it, thanks to FPM, IFX and Visual Studio Code. Let’s take a look at the structure of our very basic application.

The program keyword

The minimum amount of code for a Fortran executable is this:

program the_program
    ! Do something here
end program the_program

Comment lines start with !. To write a multi-line comment, just repeat ! at the beginning of every line.

In the previous post I included the obligatory “hello world” snippet. It would be nice if you could actually run code like that on your computer! It’s not so hard; we have very modern tooling available, and everything works equally well on Windows and Linux. The tools are freely available too.

First, install the following programs:

• Intel oneAPI Fortan Esstentials. This includes the IFX compiler.
• Fortran Package Manager (FPM). This tool is a “zero-configuration” build system which can also install and build dependencies for you.
• Visual Studio Code. We’ll use this as our editor. As far as I know there is no fully-integrated coding solution for Fortran programs. Some may use Visual Studio, some Visual Studio Code, and Jetbrains Clion may be a good option too. But none of these have solutions for all your IDE needs. After some experimenting, I think VS Code is the best option.

In a terminal, navigate to a place where you would like to create your Fortran project, and run:

program hello_world
   implicit none(type, external)

   print *, 'Hello, world!'
end program hello_world

Since January 2024 I’m working with a smart group of programmers at Deltares. They create and maintain software in the complex “business” domains of hydrodynamics, morphology, water quality and ecology. Their software is used to understand and predict all kinds of phenomena related to water, by which they “Enable Delta life”. And that’s not just for the Netherlands (many of us here live below sea level or close to rivers); the software is used around the world, by governments and businesses.

Several articles later, it turns out the little test framework has grown very fast. We’re looking at around 400 lines of code in a single test_framework module. In my opinion, this requires too much scrolling and jumping through the code to understand what’s going on or make changes. I’m aware “real-world” projects suffer from files that are a lot larger, but they are in a lot of trouble because of it. It’s smart to split large modules into smaller ones at a much earlier stage. As we discussed before, the following guidelines may be used when doing so:

The right approach to software design, in my opinion, is to work with what you have, build more functionality on top of it, then realizing there are design issues, then fixing those issues by redesigning ad hoc. Where, in my experience, most software design efforts go wrong is:

• We realize there are design issues, yet we don’t fix them. Maybe it’s too scary.
• We fix design issues, but too early, when we can’t yet know if the new design is better. We lack feedback from actual use.

In this article series, I’m happy to report, I haven’t spent too much time designing upfront. But now I want to tackle some issues, that I encountered while adding more tests. Let’s look at one of our tests:

When expanding our test suite, we’ll certainly encounter the need for an assert_equals function that works with other types than real(kind=real64). We’d want to compare values of type logical, character, integer, etc. For example, say we have a utility function str_to_upper for which we are adding a unit-test in the test_string module. This function is supposed to convert the letters in a string to their uppercase alternative. The test looks like this:

In the previous post we improved the output handling of the test runner, making output optional by using an abstract progress printer. Only our assert_equals function, which by the way should also be moved to the test_framework module, still prints directly to stdout:

function assert_equals(expected, actual, epsilon) result(assertion_failed)
   ! ... 
   assertion_failed = .false.

   if (abs(actual - expected) > epsilon) then
      print *, 'Actual: ', actual
      print *, 'Expected: ', expected
      print *, 'Epsilon: ', epsilon
      print *, 'Reals are not equal'
      assertion_failed = .true.
   end if
end function assert_equals

To fix this, we should upgrade the return type. One idea would be to return a message (variable-length string) instead of a logical, but a more flexible, future-proof alternative is to define a custom type for an error. This would allow us to pass more information back to the caller than just the message. A suitable name would be assertion_failed_t:

In the previous post we’ve successfully split the running of the tests and collecting the test results from handling those results by printing the counters and error stop-ping the test program. We still ask each test procedure to print a small description. We do this to assist the programmer when there is some kind of crash during the test run: they need to be able to find out which test caused it.

Maybe we’ve been writing more “generic” code for our custom test program than expected. We have been gradually, and quite safely, extracting code from a simple “check program”. But now that we’re looking at a pretty generic test runner that we’re going to use for all our future unit tests, we inevitably have to address the concern that we’re writing code that isn’t tested itself. Who tests the test framework? Well, the test framework itself has all the ingredients. However, testing the framework requires that we can run its code in isolation: that we can call one of its functions, and assert that it behaves correctly. Unfortunately, that isn’t possible yet. The code still lives in the main block of the tester program, and it doesn’t allow in-memory inspection of the test results, because it only prints them to the terminal, which we can’t unit-test.

In the previous post, we’ve spent some time improving a number of “temporary” tests. We introduced an assertion function to compare two real values, and we prevent failing tests from stopping the entire test program.

Moving tests to their own procedures

The approach of adding tests directly in the main test program block doesn’t really scale, as the saying goes. polyline_length may be a simple function, but for more complicated functions with multiple branches and loops, we’d have to write many of these tests. The test program keeps growing, and eventually it becomes a mess. It doesn’t help that all local variables have to be declared at the top. Even if that wasn’t needed, it isn’t very clear where a test starts or ends. Everything happens in the same scope, potentially becoming a memory management issue too. This also makes it hard to delete tests we no longer want. Removing lines from a big test program likely breaks other tests, or we forget to remove things we no longer need.

How can we know that the function we wrote, works as intended? We could run it, and manually verify its correctness. The simplest way to do this is to call the function in the main program block, print the output, and compare it with what we expect. Say our function calculates the length of a polyline, stored as a two-dimensional array of reals, representing (x,y) coordinates:

pure function polyline_length(coordinates) result(length)
   real(kind=real64), dimension(:, :), intent(in) :: coordinates

   real(kind=real64) :: length

   real(kind=real64) :: distance
   integer :: index

   length = 0.0_real64

   do index = 1, size(coordinates, 1) - 1
      distance = sqrt((coordinates(index, 1) - &
                       coordinates(index + 1, 1))**2 + &
                      (coordinates(index, 2) - &
                       coordinates(index + 1, 2))**2)
      length = length + distance
   end do
end function polyline_length

Temporary test programs

We could modify the main program block of our actual program, but it’s a lot simpler and safer to create a separate “throw-away” test program, with only the code we need. We’d write a short test program that sets up some coordinates, calls the function, then prints the result:

Unbalanced test suites

Having spoken to many developers and development teams so far, I’ve recognized several patterns when it comes to software testing. For example:

1. When the developers use a framework that encourages or sometimes even forces them to marry their code to the framework code, they only write functional tests - with a real database, making (pseudo) HTTP requests, etc. using the test tools bundled with the framework (if you’re lucky, anyway). For these teams it’s often too hard to write proper unit tests. It takes too much time, or is too difficult to set up.
2. When the developers use a framework that enables or encourages them to write code that is decoupled from the framework, they have all these nice, often pretty abstract, units of code. Those units are easy to test. What often happens is that these teams end up writing only unit tests, and don’t supply any tests or “executable specifications” proving the correctness of the behavior of the application at large.
3. Almost nobody writes proper acceptance tests. That is, most tests that use a BDD framework, are still solely focusing on the technical aspects (verifying URLs, HTML elements, rows in a database, etc.), while to be truly beneficial they should be concerned about application behavior, defined in a ubiquitous language that is used by both the technical staff and the project’s stakeholders.

Please note that these are just some very rough conclusions, there’s nothing scientific about them. It would be interesting to do some actual research though. And probably someone has already done this.

Fernando Arconada interviewed me about the subject of testing. He is writing a book about testing Symfony2 applications: Testing para Aplicaciones Symfony2. Fernando will translate this interview to Spanish and and add it to his book, together with the articles in my A better PHP testing experience series.

Who is Matthias Noback?

I’m a PHP developer, writer and speaker. I live in Zeist, The Netherlands, with my girlfriend, a son of 9 and our newborn daughter. Currently I have my own business, called Noback’s Office. This really gives me a lot of freedom: I work as a developer on one project for about half of the week and in the remaining time I can either spend some time with my family or write blog posts, or finish my second book.

Apparently not everyone agrees on how to unit test their Symfony2 controllers. Some treat controller code as the application’s “glue”: a controller does the real job of transforming a request to a response. Thus it should be tested by making a request and check the received response for the right contents. Others treat controller code just like any other code - which means that every path the interpreter may take, should be tested.

We’ve implemented filter and map operations. The results of both these operations are new arrays. Yet another common operation on arrays is the so-called reduction operation; we “reduce” multiple values to a single value. The most obvious example is calculating the sum of an array of integers: the input is an array (or list) and the output is a single integer:

pure function sum_all(numbers) result(res)
   integer, dimension(:), intent(in) :: numbers
   integer :: res

   integer :: i

   res = 0

   do i = 1, size(numbers)
      res = res + numbers(i)
   end do
end function sum_all

Just like with filter and map we can imagine a generalization of this process, which involves looping over the array elements and building up a single return value along the way. We start with the existing sum_all function. The first step is to let the user decide what the return value should be, which is the logic that happens inside the do loop. We extract a function for this:

In the previous post we looked at this dream scenario:

list([1, 3, 5, 7]) % filter(divisible_by(3)) % map(square)

As mentioned, we can’t chain function calls like this. But we can put intermediate results in local variables and achieve something similar:

integers = list([1, 3, 5, 7])
divisible_by_3 = integers%filter(divisible_by(3))
squared = divisible_by_3%map(square)

If the intermediate values are all the same type, we can reuse the variable:

integers = list([1, 3, 5, 7])
integers = integers%filter(divisible_by(3))
integers = divisible_by_3%map(square)

To enable this syntax, it’s clear that we need to introduce a derived type for a list of integers that has type-bound procedures filter and map. These functions should each return a new list, so we can call any of these functions again on the new list. Let’s start with an integer_list_t which contains an array of integers. We also provide a generic list interface with a specific procedure create_integer_list for creating the integer_list_t:

We considered the problem of filtering elements in an array, keeping only the ones we want. A decision about that is made by a function we pass as an argument.

A similar problem, but with its own challenges, is when we have an array of values and want to create a new array, with an equal number of elements, but each element has been transformed in some way. For instance, say we have an array of integers and want to square each integer. A procedural approach looks as follows:

We have a reusable filter function to which we pass the name of a procedure that matches a given interface. But can we also do something like this?

filter([1, 2, 3, 4], is_divisible_by(3))

Closures

This is a very common thing in functional programming. It involves something called higher-order functions. This means that a function may itself return a function. In this case, is_divisible_by returns a function that can be passed to filter. This returned function also needs to remember the value 3 that was passed to it. When it’s executed for each value in the integer array, it will check if that value is divisible by 3. In other languages such a function is called a closure; it’s a function that can be passed around as a value. Besides the function itself, it also has the data from the context where it was created - “bound” to it. In “ideal Fortran”, we would write something like this:

We have a working filter function for integers. Now can we generalize it? It would be nice if we could filter other types of arrays, like real arrays.

Generic filtering

As mentioned before, Fortran doesn’t have support for generics, which would allow you to use some kind of “placeholder” for types (e.g. [value_type]):

pure function filter(values, filter_func) result(res)
   [value_type], dimension(:), intent(in) :: values

   interface
      pure function filter_func(value) result(keep)
         implicit none(type, external)

         [value_type], intent(in) :: value
         logical :: keep
      end function filter_func
   end interface

   [value_type], dimension(:), allocatable :: res

   integer :: i

   res = pack(values, [(filter_func(values(i)), i=1, size(values))])
end function filter

You could then use filter on an array with any type of value, and the filter_func’s value argument would have to be of that same type, and the return type would be of that type too.

Once introduced to Fortran, we are left wondering what kind of language it really is. Traditionally, it’s been used as a procedural language, or a “Do this, do that” language as I’d like to call it. You can do some Object-Oriented programming with it, but it’s somewhat clunky, verbose, and not very “idiomatic”. (That’s not to say that you shouldn’t do it!) The same goes for Functional programming. If we’d like to write Fortran code that follows this paradigm, it becomes even more awkward. You’ll need several advanced techniques to make it work, and then there are still many limitations. The inability to chain function calls, the lack of tail call optimization and generic type-support, and the obstacles you have to conquer to get some kind of delayed execution; all of these sound like good reasons for not-even-trying…

As was the case with the SymfonyConsoleForm library, I realized I had not taken the time to properly introduce another library I created some time ago: ConvenientImmutability. This one is a lot simpler, but from a technical standpoint I think it might be interesting to at least talk a bit about the considerations that went into it.

For starters, it’s important to keep in mind that I don’t think you’ll actually need to use this library. Of course, I’ll explain why. But first: what is ConvenientImmutability about?

In the previous post we properly encapsulated the integer level with a log_level_t derived type, achieving type safety.

Now, let’s consider the following feature request: we want to show the log level in front of the message. We could do it with decoration of course, but in the scope of this article, let’s do it in the log() subroutine directly. In this case we’re “lucky” that we can access the module-private data component level, so we can do a select case on it to determine what string should be printed:

In the previous post we introduced a derived type log_level_t. The type of data passed to the log() procedure is still an integer though:

subroutine log(message, level)
   character(len=*), intent(in) :: message
   integer, intent(in) :: level

   ! ...
end subroutine log

This doesn’t stop anyone from passing a completely meaningless integer value as the level argument to log().

Increasing type-safety

We’d like to increase type-safety and the way to do it is by using a more specific type. In our case, it would be great if we could use log_level_t as the argument type for level:

In the post about decoration we declared log levels as follows:

module logging_base
   ! ...
   integer, parameter, public :: LOG_DEBUG = 0
   integer, parameter, public :: LOG_INFO = 1
   integer, parameter, public :: LOG_WARN = 2
   integer, parameter, public :: LOG_ERROR = 3
   integer, parameter, public :: LOG_FATAL = 4
   ! ...
end module logging_base

This is somewhat useful, but requires a dedicated only mention when importing each of these constants, which isn’t very nice:

program main
   use logging_base, only: log, LOG_DEBUG, LOG_WARN
    
   ! ...
   call log('A debug message', LOG_DEBUG)
    
   ! ...
   call log('A warning', LOG_WARN)
end program main

Although we could live with that, the bigger issue remains: this is not a type-safe solution. Any integer may be passed as an argument for level, and the compiler wouldn’t warn us. For example:

Fortran projects are famous for their large modules. Actually, we may also find very large files in legacy projects written in other languages, like Java, PHP, etc. These projects sometimes have files with thousands of lines of code. Fortran projects suffer more from large files I’d say, because:

• IDE support is not great. It’s often still hard to navigate to definitions or usages. There isn’t any support for automated refactorings, like extract function, move function, add parameter, etc. You often can’t get a clear overview of the structure/outline of a file.
• There are no strict indentation rules, making it easy to create a complete mess. Not everyone uses code formatting tools. Code formatting tools that exist and are used, aren’t always very good or easily configurable.
• Fortran code can be hard to read, partially because a lack of curly braces. It’s hard to visually recognize blocks of code. Another reason is that commonly there’s no distinction in letter casing between type names, function names, variable names, etc. Other languages may use CamelCase for types, CAPS for constants, pascalCase for methods, snake_case for functions, etc. In Fortran code, it’s all snake_case.

If there’s anything we can do to make Fortran code easier to deal with, it’s to split the code into smaller parts. We need to have many more modules, each with distinct topics and concerns, so we can get a quick overview of the application’s domain and capabilities. Additionally, this will help us find the right place when we want to make changes to the code.

In the previous post we saw how to use an abstraction to compose an aggregation of services of that same abstraction. There we simply delegated a call, adding no specific behavior. But we might as well add some new behavior before or after delegating the call.

As an example, let’s try to implement a new feature in the logging module: we want to add a timestamp in front of each message, so the output becomes something like this:

With our new service abstraction logger_t we are able to easily implement more features for our logger. We can do this without affecting current user code, because their code relies only on the abstraction, never on concrete implementations.

A terminal output logger

One thing we can do is define a second logger type, one that prints each message to the terminal output (stdout). Being a logger, it should also extend the abstract logger_t type, and provide an implementation for log():

With the new file_logger_t type we were able to model a logger service as a derived type. Now it’s time to make the service abstract. That is, to leave out the implementation detail that it’s a file logger (not a terminal output logger, etc.). Users should be able to use this abstract service to log any message, to whatever the application has been configured to log to.

An abstract derived type

Allowing a user to depend on an abstract service requires the use of an abstract type. The user can import this abstract type in their module or program. Meanwhile, the factory function for the abstract logger will actually return the concrete file logger, which extends the abstract logger. For object-oriented programmers this may be a familiar approach, but it’s not a very common Fortran practice. In code it looks as follows:

We’ve seen how to define a derived type to represent a point in 2D space. We were able to add a type-bound procedure to it, and finally to make its components private and define a custom constructor for it.

In essence there are two categories of derived types:

1. Derived types that hold some data. Their type-bound procedures are solely dealing with this data, and may produce valuable information with it. The point is an example of such a derived type: we can use its data to make geometrical calculations, or we can combine it together with other points into lines, polygons, etc. This all happens in memory. In typical business applications you would consider entities, value objects and data transfer objects (DTOs) to be in this first category.
2. Derived types that can perform some task that deals with things external to the application, like a file, the terminal, a network connection, etc. Such a type doesn’t hold data inside. It is able to fetch data, or send/store data. In business applications such a type is often called a service.

In object-oriented programming, service types usually offer an abstraction. This allows the client to decouple from its specific implementation details. For example, take a logger service; it’s often beneficial for the user that they don’t have to be concerned about where the logs are written to (file, screen, log aggregation server, etc.). The user only wants to call a simple log() function to log something to whatever has been configured at some higher level. Introducing a logger abstraction will take a few posts from here, but it’s good to know that service abstraction is the goal.

In the previous post we have worked on the point_t derived type, turning the module procedure distance into a type-bound procedure:

module geometry
   ! ...

   type :: point_t
      real :: x
      real :: y
   contains
      procedure :: distance => point_distance
   end type point_t

contains

   pure function point_distance(point_1, point_2) result(the_distance)
      class(point_t), intent(in) :: point_1
      
      ! ...
   end function point_distance

end module geometry

Private data components

Type-bound procedures can help us tie relevant behaviors (procedures) to derived types. It also allows us to let the derived type keep its data to itself. This is often called data hiding, or encapsulation of state. Nothing outside the module where the derived type is defined will have (read or write) access to its data components. This can be accomplished by adding the attribute private to each data component:

In the previous post we defined a derived type point_t with real data components x and y, to represent a point in 2D space. We also wrote a function to calculate the distance between two such points:

module geometry
   implicit none(type, external)

   private
   public :: point_t
   public :: distance

   type :: point_t
      real :: x
      real :: y
   end type point_t

contains

   pure function distance(point_1, point_2) result(the_distance)
      type(point_t), intent(in) :: point_1
      type(point_t), intent(in) :: point_2
      real :: the_distance

      the_distance = sqrt((point_2%x - point_1%x)**2 + &
                          (point_2%y - point_1%y)**2)
   end function distance

end module geometry

This allowed us to pass point_t instances as function arguments to distance, in another module or in the program:

We’ve seen types and variables and functions and subroutines. Now we can get to the next level and combine both variables and procedures in a so-called derived type.

Declaring a derived type and its data components

Object-oriented programming languages have classes. It took me some time to understand that classes can be seen as “extensions of the type system”. There are often primitive types like strings, integers, etc. and we can group them as properties of a class. Thereby, the class becomes some kind of advanced, composite type.

In the previous post we properly encapsulated the integer level with a log_level_t derived type, achieving type safety.

Now, let’s consider the following feature request: we want to show the log level in front of the message. We could do it with decoration of course, but in the scope of this article, let’s do it in the log() subroutine directly. In this case we’re “lucky” that we can access the module-private data component level, so we can do a select case on it to determine what string should be printed:

In the previous post we introduced a derived type log_level_t. The type of data passed to the log() procedure is still an integer though:

subroutine log(message, level)
   character(len=*), intent(in) :: message
   integer, intent(in) :: level

   ! ...
end subroutine log

This doesn’t stop anyone from passing a completely meaningless integer value as the level argument to log().

Increasing type-safety

We’d like to increase type-safety and the way to do it is by using a more specific type. In our case, it would be great if we could use log_level_t as the argument type for level:

In the post about decoration we declared log levels as follows:

module logging_base
   ! ...
   integer, parameter, public :: LOG_DEBUG = 0
   integer, parameter, public :: LOG_INFO = 1
   integer, parameter, public :: LOG_WARN = 2
   integer, parameter, public :: LOG_ERROR = 3
   integer, parameter, public :: LOG_FATAL = 4
   ! ...
end module logging_base

This is somewhat useful, but requires a dedicated only mention when importing each of these constants, which isn’t very nice:

program main
   use logging_base, only: log, LOG_DEBUG, LOG_WARN
    
   ! ...
   call log('A debug message', LOG_DEBUG)
    
   ! ...
   call log('A warning', LOG_WARN)
end program main

Although we could live with that, the bigger issue remains: this is not a type-safe solution. Any integer may be passed as an argument for level, and the compiler wouldn’t warn us. For example:

In the previous post we introduced a derived type log_level_t. The type of data passed to the log() procedure is still an integer though:

subroutine log(message, level)
   character(len=*), intent(in) :: message
   integer, intent(in) :: level

   ! ...
end subroutine log

This doesn’t stop anyone from passing a completely meaningless integer value as the level argument to log().

Increasing type-safety

We’d like to increase type-safety and the way to do it is by using a more specific type. In our case, it would be great if we could use log_level_t as the argument type for level:

Fortran projects are famous for their large modules. Actually, we may also find very large files in legacy projects written in other languages, like Java, PHP, etc. These projects sometimes have files with thousands of lines of code. Fortran projects suffer more from large files I’d say, because:

• IDE support is not great. It’s often still hard to navigate to definitions or usages. There isn’t any support for automated refactorings, like extract function, move function, add parameter, etc. You often can’t get a clear overview of the structure/outline of a file.
• There are no strict indentation rules, making it easy to create a complete mess. Not everyone uses code formatting tools. Code formatting tools that exist and are used, aren’t always very good or easily configurable.
• Fortran code can be hard to read, partially because a lack of curly braces. It’s hard to visually recognize blocks of code. Another reason is that commonly there’s no distinction in letter casing between type names, function names, variable names, etc. Other languages may use CamelCase for types, CAPS for constants, pascalCase for methods, snake_case for functions, etc. In Fortran code, it’s all snake_case.

If there’s anything we can do to make Fortran code easier to deal with, it’s to split the code into smaller parts. We need to have many more modules, each with distinct topics and concerns, so we can get a quick overview of the application’s domain and capabilities. Additionally, this will help us find the right place when we want to make changes to the code.

We’ve looked at the program and module blocks that can hold our code. Now we’ll find out where to put our data.

Declaration comes first

In Fortran, every time we want to store some value in a variable, we have to explicitly declare the type and the name of the variable first. This has to happen before any other executable statement, but after imports and other declarations like the implicit none statement:

We have a basic Fortran application and a working environment to edit, compile and run it, thanks to FPM, IFX and Visual Studio Code. Let’s take a look at the structure of our very basic application.

The program keyword

The minimum amount of code for a Fortran executable is this:

program the_program
    ! Do something here
end program the_program

Comment lines start with !. To write a multi-line comment, just repeat ! at the beginning of every line.

In the previous post I included the obligatory “hello world” snippet. It would be nice if you could actually run code like that on your computer! It’s not so hard; we have very modern tooling available, and everything works equally well on Windows and Linux. The tools are freely available too.

First, install the following programs:

• Intel oneAPI Fortan Esstentials. This includes the IFX compiler.
• Fortran Package Manager (FPM). This tool is a “zero-configuration” build system which can also install and build dependencies for you.
• Visual Studio Code. We’ll use this as our editor. As far as I know there is no fully-integrated coding solution for Fortran programs. Some may use Visual Studio, some Visual Studio Code, and Jetbrains Clion may be a good option too. But none of these have solutions for all your IDE needs. After some experimenting, I think VS Code is the best option.

In a terminal, navigate to a place where you would like to create your Fortran project, and run:

In the previous post I included the obligatory “hello world” snippet. It would be nice if you could actually run code like that on your computer! It’s not so hard; we have very modern tooling available, and everything works equally well on Windows and Linux. The tools are freely available too.

First, install the following programs:

• Intel oneAPI Fortan Esstentials. This includes the IFX compiler.
• Fortran Package Manager (FPM). This tool is a “zero-configuration” build system which can also install and build dependencies for you.
• Visual Studio Code. We’ll use this as our editor. As far as I know there is no fully-integrated coding solution for Fortran programs. Some may use Visual Studio, some Visual Studio Code, and Jetbrains Clion may be a good option too. But none of these have solutions for all your IDE needs. After some experimenting, I think VS Code is the best option.

In a terminal, navigate to a place where you would like to create your Fortran project, and run:

Get this book on Leanpub

A couple of weeks ago, Tomas Votruba emailed me saying that he just realized that we hadn’t published an update of the book we wrote together since December 2021. The book I’m talking about is “Rector - The Power of Automated Refactoring”. Two years have passed since we published the current version. Of course, we’re all very busy, but no time for excuses - this is a book about keeping projects up-to-date with almost no effort… We are meant to set an example here!

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

In October 2020 I asked Tomáš Votruba, the mastermind behind Rector, if we could have a little chat about this tool. I wanted to learn more about it and had spent a couple of days experimenting with it. Tomáš answered all my questions, which was tremendously valuable to me personally. When this happens I normally feel the need to share: there should be some kind of artefact that can be published, so others can also learn about Rector and how to extend it based on your own refactoring needs.

Get this book on Leanpub

A couple of weeks ago, Tomas Votruba emailed me saying that he just realized that we hadn’t published an update of the book we wrote together since December 2021. The book I’m talking about is “Rector - The Power of Automated Refactoring”. Two years have passed since we published the current version. Of course, we’re all very busy, but no time for excuses - this is a book about keeping projects up-to-date with almost no effort… We are meant to set an example here!

Refactoring without tests should be fine. Why is it not? When could it be safe?

From the cover of “Refactoring” by Martin Fowler:

Refactoring is a controlled technique for improving the design of an existing code base. Its essence is applying a series of small behavior-preserving transformations, each of which “too small to be worth doing”. However the cumulative effect of each of these transformations is quite significant.

Software development seems to be about change: the business changes and we need to reflect those changes, so the requirements or specifications change, frameworks and libraries change, so we have to change our integrations with them, etc. Changing the code base accordingly is often quite painful, because we made it resistant to change in many ways.

Code that resists change

I find that not every developer notices the “pain level” of a change. As an example, I consider it very painful if I can’t rename a class, or change its namespace. One reason could be that some classes aren’t auto-loaded with Composer, but are still manually loaded with require statements. Another reason could be that the framework expects the class to have a certain name, be in a certain namespace, and so on. This may be something you personally don’t consider painful, since you can avert the pain by simply not considering to rename or move classes.

It’s an extremely common problem in legacy code bases: a new way of doing things was introduced before the team decided on a way to get the old thing out.

Famous examples are:

• Introducing Doctrine ORM next to Propel
• Introducing Symfony FrameworkBundle while still using Zend controllers
• Introducing Twig for the new templates, while using Smarty for the old ones
• Introducing a Makefile while the rest of the project still uses Phing

And so on… I’m sure you also have plenty examples to add here!

TLDR;

Rector - The Power of Automated Refactoring is now 100% completed

Tomas Votruba and I first met a couple of years ago at one of my favorite conferences; the Dutch PHP Conference in Amsterdam (so actually, we’re very close to our anniversary, Tomas!). He presented Rector there and it was really inspiring. A year later I was working on a legacy migration problem: our team wanted to migrate from Doctrine ORM to “ORM-less”, with handwritten mapping code, etc. I first tried Laminas Code, a code generation tool, but it lacked many features, and also the precision that I needed. Suddenly I recalled Rector, and decided to give it a try. After some experimenting, everything worked and I learned that this tool really is amazingly powerful!

In October 2020 I asked Tomáš Votruba, the mastermind behind Rector, if we could have a little chat about this tool. I wanted to learn more about it and had spent a couple of days experimenting with it. Tomáš answered all my questions, which was tremendously valuable to me personally. When this happens I normally feel the need to share: there should be some kind of artefact that can be published, so others can also learn about Rector and how to extend it based on your own refactoring needs.

You’ve picked a good refactoring goal. You are prepared to stop the project at anytime. Now how to determine the steps that lead to the goal?

Bottom-up development

There is an interesting similarity between refactoring projects, and regular projects, where the goal is to add some new feature to the application. When working on a feature, I’m always happy to jump right in and think about what value objects, entities, controllers, etc. I need to build. Once I’ve written all that code and I’m ready to connect the dots, I often realize that I have created building blocks that I don’t even need, or that don’t offer a convenient API. This is the downside of what’s commonly called “bottom-up development”. Starting to build the low-level stuff, you can’t be certain if you’re contributing to the higher-level goal you have in mind.

Refactoring is often mentioned in the context of working with legacy code. Maybe you like to define legacy code as code without tests, or code you don’t understand, or even as code you didn’t write. Very often, legacy code is code you just don’t like, whether you wrote it, or someone else did. Since the code was written the team has introduced new and better ways of doing things. Unfortunately, half of the code base still uses the old and deprecated way…

Refactoring projects

A common case of refactoring-gone-wrong is when refactoring becomes a large project in a branch that can never be merged because the refactoring project is never completed. The refactoring project is considered a separate project, and soon starts to feel like “The Big Rewrite That Always Fails” from programming literature.

The work happens in a branch because people actually fear the change. They want to see it before they believe it, and review every single part of it before it can be merged. This process may take months. Meanwhile, other developers keep making changes to the main branch, so merging the refactoring branch is going to be a very tedious, if not dangerous thing to do. A task that, on its own, can cause the failure of the refactoring project itself.

In “Negative Architecture”, Michael Feathers speaks about certain aspects of software architecture leading to some kind of negative architecture. Feathers mentions the IO Monad from Haskell (functional programming) as an example, but there are parallel examples in object-oriented programming. For example, by using layers and the dependency inversion principle you can “guarantee” that a certain class, in a certain layer (e.g. domain, application) won’t do any IO - no talking to a database, no HTTP requests to some remote service, etc.

Statically fetching dependencies

I’ve worked with several code bases that were littered with calls to Zend_Registry::get(), sfContext::getInstance(), etc. to fetch a dependency when needed. I’m a little afraid to mention façades here, but they also belong in this list. The point of this article is not to bash a certain framework (they are all lovely), but to show how to get rid of these “centralized dependency managers” when you need to. The characteristics of these things are:

I find it very curious that legacy (PHP) code often has the following characteristics:

1. Classes with the name of a central domain concept have grown too large.
2. Methods in these classes have become very generic.

Classes grow too large

I think the following happened:

The original developers tried to capture the domain logic in these classes. They implemented it based on what they knew at the time. Other developers, who worked on the code later, had to implement new features, or modify domain logic, because, well, things change. Also, because we need more things.

This article continues where Unary call sites and intention-revealing interfaces ended.

While reading David West’s excellent book “Object Thinking”, I stumbled across an interesting quote from David Parnas on the programming method that most of us use by default:

The easiest way to describe the programming method used in most projects today was given to me by a teacher who was explaining how he teaches programming. “Think like a computer,” he said. He instructed his students to begin by thinking about what the computer had to do first and to write that down. They would then think about what the computer had to do next and continue in that way until they had described the last thing the computer would do… […]

Call sites

One of the features I love most about my IDE is the button “Find Usages”. It is invaluable when improving a legacy code base. When used on a class it will show you where this class is used (as a parameter type, in an import statement, etc.). When used on a method, it will show you where this method gets called. Users of a method are often called “clients”, but when we use “Find Usages”, we might as well use the more generic term “call sites”.

Setting the stage: Code complexity

Code complexity often gets measured by calculating the Cyclomatic Complexity per unit of code. The number can be calculated by taking all the branches of the code into consideration.

Code complexity is an indicator for several things:

• How hard it is to understand a piece of code; a high number indicates many branches in the code. When reading the code, a programmer has to keep track of all those branches, in order to understand all the different ways in which the code can work.
• How hard it is to test that piece of code; a high number indicates many branches in the code, and in order to fully test the piece of code, all those branches need to be covered separately.

In both cases, high code complexity is a really bad thing. So, in general, we always strive for low code complexity. Unfortunately, many projects that you’ll inherit (“legacy projects”), will contain code that has high code complexity, and no tests. A common hypothesis is that a high code complexity arises from a lack of tests. At the same time, it’s really hard to write tests for code with high complexity, so this is a situation that is really hard to get out.

CQRS - not a complicated thing

CQRS has some reputation issues. Mainly, people will feel that it’s too complicated to apply in their current projects. It will often be considered over-engineering. I think CQRS is simply misunderstood, which is the reason many people will not choose it as a design technique. One of the common misconceptions is that CQRS always goes together with event sourcing, which is indeed more costly and risky to implement.

Leaving Qandidate, off to Coolblue

After I had a very interesting conversation with the developers behind the Broadway framework for CQRS and event sourcing the day wasn’t over for me yet. I walked about one kilometer to the north to meet Paul de Raaij, who is a senior developer at Coolblue, a company which sells and delivers all kinds of - mostly - electrical consumer devices. Their headquarters are very close to the new and shiny Rotterdam Central station. The company itself is doing quite well. With 1000+ employees they keep needing more office space.

Legacy code

I’m happy that I’ve discovered the work of P.M. Jones recently. His and mine interests seem to align at several interesting points. Though I don’t personally enjoy “putting .308 holes in targets at 400 yards” (as quoted by Phil Sturgeon), I do care a great deal about package coupling (and cohesion for that matter) and I’m also lightly inflammable when it comes to the use of service locators. It also appears that Paul, just like myself and many others in this business, has felt the pain of working on a legacy PHP application, trying to add features to it, or change existing behavior. This is a particularly hard thing to do and because so many incompetent developers have been creating PHP applications since the dawn of PHP, chances are that a big part of the job of competent PHP developers nowadays consists of maintaining these dumpsites of include statements and superglobals.

In my previous post, I wrote about wrapping a legacy application in Silex, using output buffering and Twig. Finally, to allow for better decoupling as well as lazy loading of services, we passed the actual Silex\Application instance as the first argument of legacy controllers.

The first and quite easy way we can enhance our legacy application, is to make use of the request service (which contains all the details about the current request, wrapped inside the Symfony HttpFoundation’s Request class). So, instead of reading directly from $_GET and $_POST, we can change the edit_category() controller into the following:

Ever since I am using the Symfony Framework (be it version 1 or 2), I tend to describe every other project I’ve done (including those that were built on top of some third party “framework” like Joomla or WordPress) as a “legacy project”. Though this has sometimes felt like treason, I still keep doing it: the quality of applications written using Symfony is usually so much higher in terms of maintainability, security and code cleanliness, that even a project done last year using “only PHP” looks like a mess and seems to be no good software at all. So I feel the strong urge to rebuild everything I have in portfolio (as do many other developers), but “this time, I will do it the right way”.

Get this book on Leanpub

A couple of weeks ago, Tomas Votruba emailed me saying that he just realized that we hadn’t published an update of the book we wrote together since December 2021. The book I’m talking about is “Rector - The Power of Automated Refactoring”. Two years have passed since we published the current version. Of course, we’re all very busy, but no time for excuses - this is a book about keeping projects up-to-date with almost no effort… We are meant to set an example here!

When you’re learning to write custom rules for PHPStan or Rector, you’ll have to learn more about the PHP programming language as well. To be more precise, about the way the interpreter parses PHP code. The result of parsing PHP code is a tree of nodes which represents the structure of the code, e.g. you’ll have a Class definition node, a Method definition node, and within those method Statement nodes, and so on. Each node can be checked for errors (with PHPStan), or automatically refactored in some way (with Rector).

TLDR;

Rector - The Power of Automated Refactoring is now 100% completed

Tomas Votruba and I first met a couple of years ago at one of my favorite conferences; the Dutch PHP Conference in Amsterdam (so actually, we’re very close to our anniversary, Tomas!). He presented Rector there and it was really inspiring. A year later I was working on a legacy migration problem: our team wanted to migrate from Doctrine ORM to “ORM-less”, with handwritten mapping code, etc. I first tried Laminas Code, a code generation tool, but it lacked many features, and also the precision that I needed. Suddenly I recalled Rector, and decided to give it a try. After some experimenting, everything worked and I learned that this tool really is amazingly powerful!

In October 2020 I asked Tomáš Votruba, the mastermind behind Rector, if we could have a little chat about this tool. I wanted to learn more about it and had spent a couple of days experimenting with it. Tomáš answered all my questions, which was tremendously valuable to me personally. When this happens I normally feel the need to share: there should be some kind of artefact that can be published, so others can also learn about Rector and how to extend it based on your own refactoring needs.

Get this book on Leanpub

A couple of weeks ago, Tomas Votruba emailed me saying that he just realized that we hadn’t published an update of the book we wrote together since December 2021. The book I’m talking about is “Rector - The Power of Automated Refactoring”. Two years have passed since we published the current version. Of course, we’re all very busy, but no time for excuses - this is a book about keeping projects up-to-date with almost no effort… We are meant to set an example here!

Refactoring without tests should be fine. Why is it not? When could it be safe?

From the cover of “Refactoring” by Martin Fowler:

Refactoring is a controlled technique for improving the design of an existing code base. Its essence is applying a series of small behavior-preserving transformations, each of which “too small to be worth doing”. However the cumulative effect of each of these transformations is quite significant.

Software development seems to be about change: the business changes and we need to reflect those changes, so the requirements or specifications change, frameworks and libraries change, so we have to change our integrations with them, etc. Changing the code base accordingly is often quite painful, because we made it resistant to change in many ways.

Code that resists change

I find that not every developer notices the “pain level” of a change. As an example, I consider it very painful if I can’t rename a class, or change its namespace. One reason could be that some classes aren’t auto-loaded with Composer, but are still manually loaded with require statements. Another reason could be that the framework expects the class to have a certain name, be in a certain namespace, and so on. This may be something you personally don’t consider painful, since you can avert the pain by simply not considering to rename or move classes.

TLDR;

Rector - The Power of Automated Refactoring is now 100% completed

Tomas Votruba and I first met a couple of years ago at one of my favorite conferences; the Dutch PHP Conference in Amsterdam (so actually, we’re very close to our anniversary, Tomas!). He presented Rector there and it was really inspiring. A year later I was working on a legacy migration problem: our team wanted to migrate from Doctrine ORM to “ORM-less”, with handwritten mapping code, etc. I first tried Laminas Code, a code generation tool, but it lacked many features, and also the precision that I needed. Suddenly I recalled Rector, and decided to give it a try. After some experimenting, everything worked and I learned that this tool really is amazingly powerful!

In October 2020 I asked Tomáš Votruba, the mastermind behind Rector, if we could have a little chat about this tool. I wanted to learn more about it and had spent a couple of days experimenting with it. Tomáš answered all my questions, which was tremendously valuable to me personally. When this happens I normally feel the need to share: there should be some kind of artefact that can be published, so others can also learn about Rector and how to extend it based on your own refactoring needs.

You’ve picked a good refactoring goal. You are prepared to stop the project at anytime. Now how to determine the steps that lead to the goal?

Bottom-up development

There is an interesting similarity between refactoring projects, and regular projects, where the goal is to add some new feature to the application. When working on a feature, I’m always happy to jump right in and think about what value objects, entities, controllers, etc. I need to build. Once I’ve written all that code and I’m ready to connect the dots, I often realize that I have created building blocks that I don’t even need, or that don’t offer a convenient API. This is the downside of what’s commonly called “bottom-up development”. Starting to build the low-level stuff, you can’t be certain if you’re contributing to the higher-level goal you have in mind.

Refactoring is often mentioned in the context of working with legacy code. Maybe you like to define legacy code as code without tests, or code you don’t understand, or even as code you didn’t write. Very often, legacy code is code you just don’t like, whether you wrote it, or someone else did. Since the code was written the team has introduced new and better ways of doing things. Unfortunately, half of the code base still uses the old and deprecated way…

Refactoring projects

A common case of refactoring-gone-wrong is when refactoring becomes a large project in a branch that can never be merged because the refactoring project is never completed. The refactoring project is considered a separate project, and soon starts to feel like “The Big Rewrite That Always Fails” from programming literature.

The work happens in a branch because people actually fear the change. They want to see it before they believe it, and review every single part of it before it can be merged. This process may take months. Meanwhile, other developers keep making changes to the main branch, so merging the refactoring branch is going to be a very tedious, if not dangerous thing to do. A task that, on its own, can cause the failure of the refactoring project itself.

In the first and second part of this series we’ve been doing quite a bit of work to separate concerns that were once all combined into one function.

The major “players” in the field have been identified: there is an HttpClient and a Cache, used by different implementations of CatApi to form a testable, performing client of The Cat Api.

Representing data

We have been looking at behavior, and the overall structure of the code. But we didn’t yet look at the data that is being passed around. Currently everything is a string, including the return value of CatApi::getRandomImage(). When calling the method on an instance we are “guaranteed” to retrieve a string. I say “guaranteed” since PHP would allow anything to be returned, an object, a resource, an array, etc. Though in the case of RealCatApi::getRandomImage() we can be sure that it is a string, because we explicitly cast the return value to a string, we can’t be sure it will be useful to the caller of this function. It might be an empty string, or a string that doesn’t contain a URL, like 'I am not a URL'.

The world is not a safe thing to depend upon

When you’re running unit tests, you don’t want the world itself to be involved. Executing actual database queries, making actual HTTP requests, writing actual files, none of this is desirable: it will make your tests very slow as well as unpredictable. If the server you’re sending requests to is down, or responds in unexpected ways, your unit test will fail for the wrong reasons. A unit test should only fail if your code doesn’t do what it’s supposed to do.

Some time ago I tweeted this:

I didn't mention this yet, but I'm working on a series of videos about the subject matter of my Principles of Package Design book.

— Matthias Noback (@matthiasnoback) May 14, 2015

It turned out, creating a video tutorial isn’t working well for me. I really like writing, and speaking in public, but I’m not very happy about recording videos. I almost never watch videos myself as well, so… the video tutorial I was talking about won’t be there. Sorry!

It’s quite ironic that my most “popular” tweet has been posted while Twitter itself is in such a chaotic phase. It’s also quite ironic that I try to provide helpful suggestions for doing a better job as a programmer, yet such a bitter tweet ends up to be so popular.

Twitter and Mastodon are micro-blogging platforms. The problem with micro-blogs, and with short interactions in general, is that everybody can proceed to project onto your words whatever they like. So at some point I often feel the need to explain myself with more words, in an “actual” blog like this one.

Software development seems to be about change: the business changes and we need to reflect those changes, so the requirements or specifications change, frameworks and libraries change, so we have to change our integrations with them, etc. Changing the code base accordingly is often quite painful, because we made it resistant to change in many ways.

Code that resists change

I find that not every developer notices the “pain level” of a change. As an example, I consider it very painful if I can’t rename a class, or change its namespace. One reason could be that some classes aren’t auto-loaded with Composer, but are still manually loaded with require statements. Another reason could be that the framework expects the class to have a certain name, be in a certain namespace, and so on. This may be something you personally don’t consider painful, since you can avert the pain by simply not considering to rename or move classes.

“As I’m becoming a more experienced programmer, I tend to prefer simple solutions.” Or something similar. As is the case with many programming-related quotes, this is somewhat of a blanket statement because who doesn’t prefer simple solutions? To make it a powerful statement again, you’d have to explain what a simple solution is, and how you distinguish it from not-so-simple solutions. So the million-dollar question is “What is a simple solution?”, and I’ll answer it now.

I write about design rules a lot, but I sometimes forget to:

1. Mention that these rules can’t always be applied,
2. Describe when that would be the case, and
3. Add examples of situations where the rule really doesn’t matter.

The rules should work in most cases, but sometimes need to be “violated”. Which is too strong a word anyway. When someone points out to me that I violated a rule, I’m like: Wow! I violated the rule? I’m so sorry! Let’s fix this immediately. Whereas in practice it should be more like: Yeah, I know that rule, but it makes more sense to follow that other rule here, because […]. In other words, pointing out that a certain rule has been violated should not be a sufficient reason to adhere to that rule. My favorite example is “But that violates SRP!” (Single Responsibility Principle). Whoops, I wouldn’t want to do that! Or would I?

One of my pet peeves is using the database schema to validate data.

Several ways in which this normally happens:

• Specifying a column as “required”, e.g. email VARCHAR(255) NOT NULL
• Adding an index to force column values to be unique (e.g. CREATE UNIQUE INDEX email_idx ON users(email))
• Adding an index for foreign key integrity, including cascading deletes, etc.

Yes, I want data integrity too. No, I don’t want to rely on the database for that.

In part 1 of this short series (it’s going to end with this article) we covered how you can test-drive the queries in a repository class. Returning query results is only part of the job of a repository though. The other part is to store objects (entities), retrieve them using something like a save() and a getById() method, and possibly delete them. Some people will implement these two jobs in one repository class, some like to use two or even many repositories for this. When you have a separate write and read model (CQRS), the read model repositories will have the querying functionality (e.g. find me all the active products), the write model repositories will have the store/retrieve/delete functionality.

There’s something I’ve only been doing since a year or so, and I’ve come to like it a lot. My previous testing experiences were mostly at the level of unit tests or functional/system tests. What was left out of the equation was integration tests. The perfect example of which is a test that proves that your custom repository class works well with the type of database that you use for the project, and possibly the ORM or database abstraction library you use to talk with that database. A test for a repository can’t be a unit test; that wouldn’t make sense. You’d leave a lot of assumptions untested. So, no mocking is allowed.

I recently wrote about when to add an interface to a class. After explaining good reasons for adding an interface, I claim that if none of those reasons apply in your situation, you should just use a class and declare it “final”.

PHP 5 introduces the final keyword, which prevents child classes from overriding a method by prefixing the definition with final. If the class itself is being defined final then it cannot be extended.

Recently I read a comment on Twitter by Nikola Poša:

I guess the discussion on my thread is going in the wrong direction because I left out a crucial hashtag: #NoCommentsInCode - avoid comments in code, write descriptive classes, methods, variables.https://t.co/MuHoOFXCvV

— Nikola Poša (@nikolaposa) July 13, 2018

He was providing us with a useful suggestion, one that I myself have been following ever since reading “Clean Code” by Robert Martin. The paraphrased suggestion in that book, as well as in the tweet, is to consider a comment to be a naming issue in disguise, and to solve that issue, instead of keeping the comment. By the way, the book has some very nice examples of how comments should and should not be used.

In the previous article we discussed a sensible layer system, consisting of three layers:

• Domain
• Application
• Infrastructure

Infrastructure

The infrastructure layer, containing everything that connects the application’s use cases to “the world outside” (like users, hardware, other applications), can become quite large. As I already remarked, a lot of our software consists of infrastructure code, since that’s the realm of things complicated and prone to break. Infrastructure code connects our precious clean code to:

The first key concept of what I think is a very simple, at the very least “clean” architecture, is the concept of a layer. A layer itself is actually nothing, if you think about it. It’s simply determined by how it’s used. Let’s stay a bit philosophical, before we dive into some concrete architectural advice.

Qualities of layers

A layer in software serves the following (more or less abstract) purposes:

Looking back at my old blog posts, I think it’s good to write down a more balanced view on application architecture than the one that speaks from some of the older posts from 2013 and 2014. Before I do, I allow myself a quick self-centered trip down memory lane.

Why Symfony? Seven facts

The archive tells an interesting story about how my thoughts about software development changed over time. It all started with my first post in October 2011, How to make your service use tags. Back in those days, version 2 of the Symfony framework was still quite young. I had been working with symfony (version 1, small caps) for a couple of years and all of a sudden I was using Symfony2, and I absolutely loved this new shiny framework. I learned as much as I could about it (just as I had done with version 1), and eventually I became a Certified Symfony Developer, at the first round of exams held in Paris, during a Symfony Live conference. During those years I blogged and spoke a lot about Symfony, contributed to the documentation, and produced several open source bundles. I also wrote a book about it: A Year With Symfony.

During a workshop we were discussing the concept of a Data Transfer Object (DTO). The main characteristic of a DTO is that it holds only primitive-type values (strings, integers, booleans), lists or maps of these values including “nested” DTOs. Not sure who came up with this idea, but I’m using it because it ensures that the DTO becomes a data structure that only enforces a schema (field names, the expected types, required fields, and optional fields), but doesn’t enforce semantics for any value put into it. That way it can be created from any data source, like submitted form values, CLI arguments, JSON, XML, Yaml, and so on. Using primitive values in a DTO makes it clear that the values are not validated. The DTO is just used to transfer or carry data from one layer to the next. A question that popped up during the workshop: can we consider DateTimeImmutable a primitive-type value too? If so, can we use this type inside DTOs?

Having discussed different aspects of simplicity in programming solutions, let’s start with the first topic that should be scrutinized regarding their simplicity: persisting model objects. As you may know, we have competing solutions which fall into two categories: they will follow either the Active Record (AR) or the Data Mapper pattern (DM) (as described in Martin Fowler’s “Patterns of Enterprise Application Architecture”, abbrev. PoEAA).

Active record

How do we recognize the AR pattern? It’s when you instantiate a model object and then call save() on it:

When to use a trait? Never.

Well, a trait could be considered to have a few benefits:

Benefits

1. If you want to reuse some code between multiple classes, using a trait is an alternative for extending the class. In that case the trait may be the better option because it doesn’t become part of the type hierarchy, i.e. a class that uses a trait isn’t “an instance of that trait”.
2. A trait can save you some manual copy/paste-ing by offering compile-time copy/paste-ing instead.

Downsides

On the other hand, there are several problems with traits. For instance:

In part 1 of this series we discussed the size and complexity metrics calculated by phploc. We continue with a discussion about dependencies and structure.

Structure

This section gives us an idea of how many things there are of a certain type. These can be useful indicators too. For example, if the number of Namespaces is low, there may be a lack of grouping, which is bad for discoverability. It’ll be hard to find out where a certain piece of logic can be found in the code. Too many namespaces, relative to the number of Classes/Interfaces/Traits, is not a good sign either. I would expect every namespace to have a couple of classes that naturally belong together.

When I want to get a very rough idea of the quality of both code and structure of a PHP code base, I like to run phploc on it. This is a tool created by Sebastian Bergmann for (I assume) exactly this purpose. It produces the following kind of output:

Directories                                          3
Files                                               10

Size
  Lines of Code (LOC)                             1882
  Comment Lines of Code (CLOC)                     255 (13.55%)
  Non-Comment Lines of Code (NCLOC)               1627 (86.45%)
  Logical Lines of Code (LLOC)                     377 (20.03%)
    Classes                                        351 (93.10%)
      Average Class Length                          35
        Minimum Class Length                         0
        Maximum Class Length                       172
      Average Method Length                          2
        Minimum Method Length                        1
        Maximum Method Length                      117
    Functions                                        0 (0.00%)
      Average Function Length                        0
    Not in classes or functions                     26 (6.90%)

Cyclomatic Complexity
  Average Complexity per LLOC                     0.49
  Average Complexity per Class                   19.60
    Minimum Class Complexity                      1.00
    Maximum Class Complexity                    139.00
  Average Complexity per Method                   2.43
    Minimum Method Complexity                     1.00
    Maximum Method Complexity                    96.00

Dependencies
  Global Accesses                                    0
    Global Constants                                 0 (0.00%)
    Global Variables                                 0 (0.00%)
    Super-Global Variables                           0 (0.00%)
  Attribute Accesses                                85
    Non-Static                                      85 (100.00%)
    Static                                           0 (0.00%)
  Method Calls                                     280
    Non-Static                                     276 (98.57%)
    Static                                           4 (1.43%)

Structure
  Namespaces                                         3
  Interfaces                                         1
  Traits                                             0
  Classes                                            9
    Abstract Classes                                 0 (0.00%)
    Concrete Classes                                 9 (100.00%)
  Methods                                          130
    Scope
      Non-Static Methods                           130 (100.00%)
      Static Methods                                 0 (0.00%)
    Visibility
      Public Methods                               103 (79.23%)
      Non-Public Methods                            27 (20.77%)
  Functions                                          0
    Named Functions                                  0 (0.00%)
    Anonymous Functions                              0 (0.00%)
  Constants                                          0
    Global Constants                                 0 (0.00%)
    Class Constants                                  0 (0.00%)

These numbers are statistics, and as you know, they can be used to tell the biggest lies. Without the context of the actual code, you can interpret them in any way you like. So you need to be careful making conclusions based on them. However, in my experience these numbers are often quite good indicators of the overall code quality as well as the structural quality of a project.

This is another excerpt from my latest book, which is currently part of Manning’s Early Access Program. Take 37% off Object Design Style Guide by entering fccnoback into the discount code box at checkout at manning.com.

Make sure to read part 1 first.

Create read models directly from their data source

Instead of creating a StockReport model from PurchaseOrderForStock objects, we could go directly to the source of the data, that is, the database where the application stores its purchase orders. If this is a relational database, there might be a table called purchase_orders, with columns for purchase_order_id, product_id, ordered_quantity, and was_received. If that’s the case, then StockReportRepository wouldn’t have to load any other object before it could build a StockReport object; it could make a single SQL query and use it to create the StockReport, as shown in Listing 11).

I’m happy to share with you an excerpt of my latest book, which is currently part of Manning’s Early Access Program. Take 37% off Object Design Style Guide by entering fccnoback into the discount code box at checkout at manning.com.

Chapter 7: Dividing responsibilities

We’ve looked at how objects can be used to retrieve information, or perform tasks. The methods for retrieving information are called query methods, the ones that perform tasks are command methods. Service objects may combine both of these responsibilities. For instance, a repository (like the one in Listing 1) could perform the task of saving an entity to the database, and at the same time it would also be capable of retrieving an entity from the database.

Please note: The Object Design Style Guide is now available via Manning,

With a foreword by Ross Tuck

Today I’ve released the revised edition of my latest book “Style Guide for Object Design”. Over the past few weeks I’ve fixed many issues, smaller and larger ones. Ross Tuck and Laura Cody have read the manuscript and provided me with excellent feedback, which has helped me a lot to get the book in a better shape. I added many sections, asides, and even an extra chapter, because some topics deserve more detail and background than just a few paragraphs. Oh, and Ross wrote the kindest of forewords to kick off the book. It’s available in the free sample of the book.

When you’re looking at a function (an actual function or a method), you can usually identify several blocks of code in there. There are pre-conditions, there’s the function body, and there may be post-conditions. The pre-conditions are there to verify that the function can safely proceed to do its real job. Post-conditions may be there to verify that you’re going to give something back to the caller that will make sense to them.

I’m currently revising my book “Principles of Package Design”. It covers lots of design principles, like the SOLID principles and the lesser known Package (or Component) Design Principles. When discussing these principles in the book, I regularly encourage the reader to add more interfaces to their classes, to make the overall design of the package or application more flexible. However, not every class needs an interface, and not every interface makes sense. I thought it would be useful to enumerate some good reasons for adding an interface to a class. At the end of this post I’ll make sure to mention a few good reasons for not adding an interface too.

In “Negative Architecture”, Michael Feathers speaks about certain aspects of software architecture leading to some kind of negative architecture. Feathers mentions the IO Monad from Haskell (functional programming) as an example, but there are parallel examples in object-oriented programming. For example, by using layers and the dependency inversion principle you can “guarantee” that a certain class, in a certain layer (e.g. domain, application) won’t do any IO - no talking to a database, no HTTP requests to some remote service, etc.

Consider the following rule:

When you create an object, it should be complete, consistent and valid in one go.

It is derived from the more general principle that it should not be possible for an object to exist in an inconsistent state. I think this is a very important rule, one that will gradually lead everyone from the swamps of those dreaded “anemic” domain models. However, the question still remains: what does all of this mean?

I’m working on another “multi-tenant” PHP web application project and I noticed an interesting series of events. It felt like a natural progression and by means of a bit of dangerous induction, I’m posing the hypothesis that this is how things are just bound to happen in such projects.

In the beginning we start out with a framework that has some authentication functionality built-in. We can get the “current user” from the session, or from some other session-based object. We’ll also need the “current company” (or the “current organization”) of which the current user is a member.

I recently came across two interesting methods that were part of a bigger class that I had to redesign:

class OrderLine
{
    /**
     * @var float
     */
    private $quantityOrdered;
    
    // ...

    /**
     * @param float $quantity
     */
    public function processDelivery($quantity)
    {
        $this->quantityDelivered += $quantity;
        
        $this->quantityOpen = $this->quantityOrdered - $quantity;
        if ($this->quantityOpen < 0) {
            $this->quantityOpen = 0;
        }
    }

    /**
     * @param float $quantity
     */
    public function undoDelivery($quantity)
    {
        $this->quantityDelivered -= $quantity;
        if ($this->quantityDelivered < 0) {
            $this->quantityDelivered = 0;
        }
        
        $this->quantityOpen += $quantity;
        if ($this->quantityOpen > $this->quantityOrdered) {
            $this->quantityOpen = $this->quantityOrdered;
        }
    }
}

Of course, I’ve already cleaned up the code here to allow you to better understand it.

This article continues where Unary call sites and intention-revealing interfaces ended.

While reading David West’s excellent book “Object Thinking”, I stumbled across an interesting quote from David Parnas on the programming method that most of us use by default:

The easiest way to describe the programming method used in most projects today was given to me by a teacher who was explaining how he teaches programming. “Think like a computer,” he said. He instructed his students to begin by thinking about what the computer had to do first and to write that down. They would then think about what the computer had to do next and continue in that way until they had described the last thing the computer would do… […]

Call sites

One of the features I love most about my IDE is the button “Find Usages”. It is invaluable when improving a legacy code base. When used on a class it will show you where this class is used (as a parameter type, in an import statement, etc.). When used on a method, it will show you where this method gets called. Users of a method are often called “clients”, but when we use “Find Usages”, we might as well use the more generic term “call sites”.

CQRS - not a complicated thing

CQRS has some reputation issues. Mainly, people will feel that it’s too complicated to apply in their current projects. It will often be considered over-engineering. I think CQRS is simply misunderstood, which is the reason many people will not choose it as a design technique. One of the common misconceptions is that CQRS always goes together with event sourcing, which is indeed more costly and risky to implement.

Last year I took several Scala courses on Coursera. It was an interesting experience and it has brought me a lot of new ideas. One of these is the idea of a singleton object (as opposed to a class). It has the following characteristics:

• There is only one instance of it (hence it’s called a “singleton”, but isn’t an implementation of the Singleton design pattern).
• It doesn’t need to be explicitly instantiated (it doesn’t have the traditional static getInstance() method). In fact, an instance already exists when you first want to use it.
• There doesn’t have to be built-in protection against multiple instantiations (as there is and can only be one instance, by definition).

Converting this notion to PHP is impossible, but if it was possible, you could do something like this:

In the first and second part of this series we’ve been doing quite a bit of work to separate concerns that were once all combined into one function.

The major “players” in the field have been identified: there is an HttpClient and a Cache, used by different implementations of CatApi to form a testable, performing client of The Cat Api.

Representing data

We have been looking at behavior, and the overall structure of the code. But we didn’t yet look at the data that is being passed around. Currently everything is a string, including the return value of CatApi::getRandomImage(). When calling the method on an instance we are “guaranteed” to retrieve a string. I say “guaranteed” since PHP would allow anything to be returned, an object, a resource, an array, etc. Though in the case of RealCatApi::getRandomImage() we can be sure that it is a string, because we explicitly cast the return value to a string, we can’t be sure it will be useful to the caller of this function. It might be an empty string, or a string that doesn’t contain a URL, like 'I am not a URL'.

The world is not a safe thing to depend upon

When you’re running unit tests, you don’t want the world itself to be involved. Executing actual database queries, making actual HTTP requests, writing actual files, none of this is desirable: it will make your tests very slow as well as unpredictable. If the server you’re sending requests to is down, or responds in unexpected ways, your unit test will fail for the wrong reasons. A unit test should only fail if your code doesn’t do what it’s supposed to do.

Some time ago I tweeted this:

I didn't mention this yet, but I'm working on a series of videos about the subject matter of my Principles of Package Design book.

— Matthias Noback (@matthiasnoback) May 14, 2015

It turned out, creating a video tutorial isn’t working well for me. I really like writing, and speaking in public, but I’m not very happy about recording videos. I almost never watch videos myself as well, so… the video tutorial I was talking about won’t be there. Sorry!

During a workshop we were discussing the concept of a Data Transfer Object (DTO). The main characteristic of a DTO is that it holds only primitive-type values (strings, integers, booleans), lists or maps of these values including “nested” DTOs. Not sure who came up with this idea, but I’m using it because it ensures that the DTO becomes a data structure that only enforces a schema (field names, the expected types, required fields, and optional fields), but doesn’t enforce semantics for any value put into it. That way it can be created from any data source, like submitted form values, CLI arguments, JSON, XML, Yaml, and so on. Using primitive values in a DTO makes it clear that the values are not validated. The DTO is just used to transfer or carry data from one layer to the next. A question that popped up during the workshop: can we consider DateTimeImmutable a primitive-type value too? If so, can we use this type inside DTOs?

In essence, everything is a string.

Well, you can always go one layer deeper and find out what a string really is, but for web apps I work on, both input data and output data are strings. The input is an HTTP request, which is a plain-text message that gets passed to the web server, the PHP server, the framework, and finally a user-land controller. The output is an HTTP response, which is also a plain-text message that gets passed to the client. If my app needs the database to load or store some data, that data too is in its initial form a string. It needs to be deserialized into objects to do something and later be serialized into strings so we can store the results.

During a workshop we were discussing the concept of a Data Transfer Object (DTO). The main characteristic of a DTO is that it holds only primitive-type values (strings, integers, booleans), lists or maps of these values including “nested” DTOs. Not sure who came up with this idea, but I’m using it because it ensures that the DTO becomes a data structure that only enforces a schema (field names, the expected types, required fields, and optional fields), but doesn’t enforce semantics for any value put into it. That way it can be created from any data source, like submitted form values, CLI arguments, JSON, XML, Yaml, and so on. Using primitive values in a DTO makes it clear that the values are not validated. The DTO is just used to transfer or carry data from one layer to the next. A question that popped up during the workshop: can we consider DateTimeImmutable a primitive-type value too? If so, can we use this type inside DTOs?

A common misunderstanding in my workshops (well, whose fault is it then? ;)), is about the distinction between a DTO and a value object. And so I’ve been looking for a way to categorize these objects without mistake.

What’s a DTO and how do you recognize it?

A DTO is an object that holds primitive data (strings, booleans, floats, nulls, arrays of these things). It defines the schema of this data by explicitly declaring the names of the fields and their types. It can only guarantee that all the data is there, simply by relying on the strictness of the programming language: if a constructor has a required parameter of type string, you have to pass a string, or you can’t even instantiate the object. However, a DTO does not provide any guarantee that the values actually make sense from a business perspective. Strings could be empty, integers could be negative, etc.

I’d like to start this article with a quote from Ross Tuck’s article “Persisting Value Objects in Doctrine”. He describes different ways of persisting value objects when using Doctrine ORM. At the end of the page he gives us the following option - the “nuclear” one:

[…] Doctrine is great for the vast majority of applications but if you’ve got edge cases that are making your entity code messy, don’t be afraid to toss Doctrine out. Setup an interface for your repositories and create an alternate implementation where you do the querying or mapping by hand. It might be a PITA but it might also be less frustration in the long run.

This is a question that always pops up during my workshops: when and where to determine the ID of an entity? There are different answers, no best answer. Well, there are two best answers, but they apply to two different situations.

Auto-incrementing IDs, by the database

Traditionally, all you need for an entity to have an ID is to designate one integer column in the database as the primary key, and mark it as “auto-incrementing”. So, once a new entity gets persisted as a record in the database (using your favorite ORM), it will get an ID. That is, the entity has no identity until it has been persisted. Even though this happens everywhere, and almost always; it’s a bit weird, because:

I recently came across two interesting methods that were part of a bigger class that I had to redesign:

class OrderLine
{
    /**
     * @var float
     */
    private $quantityOrdered;
    
    // ...

    /**
     * @param float $quantity
     */
    public function processDelivery($quantity)
    {
        $this->quantityDelivered += $quantity;
        
        $this->quantityOpen = $this->quantityOrdered - $quantity;
        if ($this->quantityOpen < 0) {
            $this->quantityOpen = 0;
        }
    }

    /**
     * @param float $quantity
     */
    public function undoDelivery($quantity)
    {
        $this->quantityDelivered -= $quantity;
        if ($this->quantityDelivered < 0) {
            $this->quantityDelivered = 0;
        }
        
        $this->quantityOpen += $quantity;
        if ($this->quantityOpen > $this->quantityOrdered) {
            $this->quantityOpen = $this->quantityOrdered;
        }
    }
}

Of course, I’ve already cleaned up the code here to allow you to better understand it.

A common misunderstanding in my workshops (well, whose fault is it then? ;)), is about the distinction between a DTO and a value object. And so I’ve been looking for a way to categorize these objects without mistake.

What’s a DTO and how do you recognize it?

A DTO is an object that holds primitive data (strings, booleans, floats, nulls, arrays of these things). It defines the schema of this data by explicitly declaring the names of the fields and their types. It can only guarantee that all the data is there, simply by relying on the strictness of the programming language: if a constructor has a required parameter of type string, you have to pass a string, or you can’t even instantiate the object. However, a DTO does not provide any guarantee that the values actually make sense from a business perspective. Strings could be empty, integers could be negative, etc.

When you’re learning to write custom rules for PHPStan or Rector, you’ll have to learn more about the PHP programming language as well. To be more precise, about the way the interpreter parses PHP code. The result of parsing PHP code is a tree of nodes which represents the structure of the code, e.g. you’ll have a Class definition node, a Method definition node, and within those method Statement nodes, and so on. Each node can be checked for errors (with PHPStan), or automatically refactored in some way (with Rector).

When you’re learning to write custom rules for PHPStan or Rector, you’ll have to learn more about the PHP programming language as well. To be more precise, about the way the interpreter parses PHP code. The result of parsing PHP code is a tree of nodes which represents the structure of the code, e.g. you’ll have a Class definition node, a Method definition node, and within those method Statement nodes, and so on. Each node can be checked for errors (with PHPStan), or automatically refactored in some way (with Rector).

This article is about a topic that didn’t make the cut for my latest book, Recipes for Decoupling. It still contains some useful ideas I think, so here it is anyway!

DateTimeImmutable is mutable

I don’t know where I first heard it, but PHP’s DateTimeImmutable is not immutable:

<?php

$dt = new DateTimeImmutable('now');
echo $dt->getTimestamp() . "\n";

$dt->__construct('tomorrow');
echo $dt->getTimestamp() . "\n";

The result:

1656927919
1656972000

Indeed, DateTimeImmutable is not really immutable because its internal state can be modified after instantiation. After calling __construct() again, any existing object reference will have the modified state as well. But an even bigger surprise might be that if a constructor is public, it’s just another method that you can call. You’re not supposed to, but you can. Which is another reason to make the constructor of non-service objects private and add a public static constructor to a class that you want to be immutable:

Having discussed different aspects of simplicity in programming solutions, let’s start with the first topic that should be scrutinized regarding their simplicity: persisting model objects. As you may know, we have competing solutions which fall into two categories: they will follow either the Active Record (AR) or the Data Mapper pattern (DM) (as described in Martin Fowler’s “Patterns of Enterprise Application Architecture”, abbrev. PoEAA).

Active record

How do we recognize the AR pattern? It’s when you instantiate a model object and then call save() on it:

Having discussed different aspects of simplicity in programming solutions, let’s start with the first topic that should be scrutinized regarding their simplicity: persisting model objects. As you may know, we have competing solutions which fall into two categories: they will follow either the Active Record (AR) or the Data Mapper pattern (DM) (as described in Martin Fowler’s “Patterns of Enterprise Application Architecture”, abbrev. PoEAA).

Active record

How do we recognize the AR pattern? It’s when you instantiate a model object and then call save() on it:

Having discussed different aspects of simplicity in programming solutions, let’s start with the first topic that should be scrutinized regarding their simplicity: persisting model objects. As you may know, we have competing solutions which fall into two categories: they will follow either the Active Record (AR) or the Data Mapper pattern (DM) (as described in Martin Fowler’s “Patterns of Enterprise Application Architecture”, abbrev. PoEAA).

Active record

How do we recognize the AR pattern? It’s when you instantiate a model object and then call save() on it:

“As I’m becoming a more experienced programmer, I tend to prefer simple solutions.” Or something similar. As is the case with many programming-related quotes, this is somewhat of a blanket statement because who doesn’t prefer simple solutions? To make it a powerful statement again, you’d have to explain what a simple solution is, and how you distinguish it from not-so-simple solutions. So the million-dollar question is “What is a simple solution?”, and I’ll answer it now.

By request: what’s my workflow for writing books? Steps, tools, etc.

Writing with the Leanpub platform

A long time ago I noticed that testing-advocate Chris Hartjes published his books on Leanpub. When I had the idea of writing a book about the Symfony framework, I tried this platform and it was a good match. You can write your book in Markdown, commit the manuscript to a GitHub repository, and push the button to publish a new version. Leanpub will generate EPUB and PDF versions for you. Readers can be updated about releases via email or a notification on the website.

By request: what’s my workflow for writing books? Steps, tools, etc.

Writing with the Leanpub platform

A long time ago I noticed that testing-advocate Chris Hartjes published his books on Leanpub. When I had the idea of writing a book about the Symfony framework, I tried this platform and it was a good match. You can write your book in Markdown, commit the manuscript to a GitHub repository, and push the button to publish a new version. Leanpub will generate EPUB and PDF versions for you. Readers can be updated about releases via email or a notification on the website.

It’s very easy not to write a book.

Mostly since it’s such an awful lot of work. You’ll first need to figure out what you’re going to write about, and find an angle that makes it interesting enough for potential readers to buy the book and spend the time to finish it. The writing itself is hard work as well, and may take several years. Along the way you’ll be telling yourself things that will keep you from finishing the task: I’m not doing anything original here. Nobody will be interested. Somebody else wrote a better book about this than I could ever do. If I don’t cover topic B as well, it’s not going to be a useful book at all.

A very important “trick” in finding the flow in life is: do what you like most. Of course, you have to do things you don’t like (and then you need different life hacks), but when you can do something you like, you’ll find that you’ll be more successful at it.

When it comes to blogging, I find that it helps to follow my instincts, to write about whatever I like to write about at the moment. I can think of a list of things that need blogging about, but I end up not writing about them because they don’t light the fire inside me (anymore).

When to use a trait? Never.

Well, a trait could be considered to have a few benefits:

Benefits

1. If you want to reuse some code between multiple classes, using a trait is an alternative for extending the class. In that case the trait may be the better option because it doesn’t become part of the type hierarchy, i.e. a class that uses a trait isn’t “an instance of that trait”.
2. A trait can save you some manual copy/paste-ing by offering compile-time copy/paste-ing instead.

Downsides

On the other hand, there are several problems with traits. For instance:

This article shows an example of framework decoupling. You’ll find a more elaborate discussion in my latest book, Recipes for Decoupling.

Why would it be nice to decouple your user model from the framework’s security user or authentication model?

Reason 1: Hexagonal architecture

I like to use hexagonal architecture in my applications, which means among other things that the entities from my domain model stay behind a port. They are never exposed to, for instance, a controller, or a template. Whenever I want to show anything to the user, I create a dedicated view model for it.

This is another excerpt from my latest book, which is currently part of Manning’s Early Access Program. Take 37% off Object Design Style Guide by entering fccnoback into the discount code box at checkout at manning.com.

Make sure to read part 1 first.

Create read models directly from their data source

Instead of creating a StockReport model from PurchaseOrderForStock objects, we could go directly to the source of the data, that is, the database where the application stores its purchase orders. If this is a relational database, there might be a table called purchase_orders, with columns for purchase_order_id, product_id, ordered_quantity, and was_received. If that’s the case, then StockReportRepository wouldn’t have to load any other object before it could build a StockReport object; it could make a single SQL query and use it to create the StockReport, as shown in Listing 11).

I’m happy to share with you an excerpt of my latest book, which is currently part of Manning’s Early Access Program. Take 37% off Object Design Style Guide by entering fccnoback into the discount code box at checkout at manning.com.

Chapter 7: Dividing responsibilities

We’ve looked at how objects can be used to retrieve information, or perform tasks. The methods for retrieving information are called query methods, the ones that perform tasks are command methods. Service objects may combine both of these responsibilities. For instance, a repository (like the one in Listing 1) could perform the task of saving an entity to the database, and at the same time it would also be capable of retrieving an entity from the database.

CQRS - not a complicated thing

CQRS has some reputation issues. Mainly, people will feel that it’s too complicated to apply in their current projects. It will often be considered over-engineering. I think CQRS is simply misunderstood, which is the reason many people will not choose it as a design technique. One of the common misconceptions is that CQRS always goes together with event sourcing, which is indeed more costly and risky to implement.

Visiting Qandidate in Rotterdam

Last Thursday I had the honor of meeting (part of the) Broadway team: three very smart developers working for Qandidate in central Rotterdam: Gediminas Šedbaras, Willem-Jan Zijderveld and Fritsjan. A short walk from the train station brought me to their office, where we talked about CQRS, event sourcing and some of the practical aspects of using the Broadway framework.

As you may have read I’ve been experimenting with Broadway extensively during these last weeks. Knowing a lot about the theory behind it, it was really nice to see how everything worked so smoothly. I had some questions however, which have now been answered (thank you guys for taking the time to do this!).

This article shows an example of framework decoupling. You’ll find a more elaborate discussion in my latest book, Recipes for Decoupling.

Why would it be nice to decouple your user model from the framework’s security user or authentication model?

Reason 1: Hexagonal architecture

I like to use hexagonal architecture in my applications, which means among other things that the entities from my domain model stay behind a port. They are never exposed to, for instance, a controller, or a template. Whenever I want to show anything to the user, I create a dedicated view model for it.

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

Years ago my co-worker Maurits introduced me to the term “magic” in programming. He also provided the valuable dichotomy of convention and configuration (or in fact, he’d choose configuration over convention…). I think this distinction could be very helpful in psychological research, figuring out why some people prefer framework X over framework Y. One requires the developer to spell out everything they want in elaborate configuration files, the other relies on convention: placing certain files with certain names and certain methods in certain places will make everything work “magically”.

This article is an excerpt from my book Advanced Web Application Architecture. It contains a couple of sections from the conclusion of Part I: Decoupling from infrastructure.

This chapter covers:

• A deeper discussion on the distinction between core and infrastructure code
• A summary of the strategy for pushing infrastructure to the sides
• A recommendation for using a domain- and test-first approach to software development
• A closer look at the concept of “pure” object-oriented programming

Core code and infrastructure code

In Chapter 1 we’ve looked at definitions for the terms core code and infrastructure code. What I personally find useful about these definitions is that you can look at a piece of code and find out if the definitions apply to it. You can then decide if it’s either core or infrastructure code. But there are other ways of applying these terms to software. One way is to consider the bigger picture of the application and its interactions with actors. You’ll find the term actor in books about user stories and use cases by authors like Ivar Jacobson and Alistair Cockburn, who make a distinction between:

As a rather unusual pastime for the Saturday night I attended the third Domain-Driven Design Africa online meetup. Thomas Pierrain a.k.a. use case driven spoke about his adaptation of Hexagonal architecture. “It’s not by the book,” as he said, but it solves a lot of the issues he encountered over the years. I’ll try to summarize his approach here, but I recommend watching the full talk as well.

Hexagonal architecture

Hexagonal architecture makes a distinction between the use cases of an application, and how they are connected to their surrounding infrastructure. Domain logic is represented by pure code (in the FP sense of the word), surrounded by a set of adapters that expose the use cases of the application to actual users and connect the application to databases, messages queues, and so on.

Since I’ve been writing a lot about decoupled application development it made sense that one of my readers asked the following question: “Why should we use a framework?” The quick answer is: because you need it. A summary of the reasons:

• It would be too much work to replace all the work that the framework does for you with code written by yourself. Software development is too costly for this.
• Framework maintainers have fixed many issues before you even encountered them. They have done everything to make the code secure, and when a new security issue pops up, they fix it so you can just pull the latest version of the framework.
• By not using a framework you will be decoupled from Symfony, Laravel, etc. but you will be coupled to Your Own Framework, which is a bigger problem since you’re the maintainer and it’s likely that you won’t actually maintain it (in my experience, this is what often happens to projects that use their own home-grown framework).

So, yes, you/we need a framework. At the same time you may want to write framework-decoupled code whenever possible.

Decoupling from a service locator - shouldn’t that be: don’t use a service locator?

Well, not really, since there are lots of valid use cases for using a service locator. The main use case is for making things lazy-loading (yes, you can also use some kind of proxy mechanism for that, but let’s assume you need something simpler). Say we have this EventDispatcher class:

class EventDispatcher
{
    public function __construct(array $listeners)
    {
        $this->listeners = $listeners;
    }

    public function dispatch($event)
    {
        foreach ($this->listeners[$event] as $listener) {
            $listener->notify();
        }
    }
}

Now it appears that some event listeners are rather expensive to instantiate. And, even though it may never be notified (because it listens to a rare event), in order to register any event listener, we need to instantiate it:

This article shows an example of framework decoupling. You’ll find a more elaborate discussion in my latest book, Recipes for Decoupling.

Why would it be nice to decouple your user model from the framework’s security user or authentication model?

Reason 1: Hexagonal architecture

I like to use hexagonal architecture in my applications, which means among other things that the entities from my domain model stay behind a port. They are never exposed to, for instance, a controller, or a template. Whenever I want to show anything to the user, I create a dedicated view model for it.

This article is an excerpt from my book Advanced Web Application Architecture. It contains a couple of sections from the conclusion of Part I: Decoupling from infrastructure.

This chapter covers:

• A deeper discussion on the distinction between core and infrastructure code
• A summary of the strategy for pushing infrastructure to the sides
• A recommendation for using a domain- and test-first approach to software development
• A closer look at the concept of “pure” object-oriented programming

Core code and infrastructure code

In Chapter 1 we’ve looked at definitions for the terms core code and infrastructure code. What I personally find useful about these definitions is that you can look at a piece of code and find out if the definitions apply to it. You can then decide if it’s either core or infrastructure code. But there are other ways of applying these terms to software. One way is to consider the bigger picture of the application and its interactions with actors. You’ll find the term actor in books about user stories and use cases by authors like Ivar Jacobson and Alistair Cockburn, who make a distinction between:

A lot can happen in 9 years. Back then I was still advocating that you should unit-test your controllers and that setter injection is very helpful when replacing controller dependencies with test doubles. I’ve changed my mind: constructor injection is the right way for any service object, including controllers. And controllers shouldn’t be unit tested, because:

• Those unit tests tend to be a one-to-one copy of the controller code itself. There is no healthy distance between the test and the implementation.
• Controllers need some form of integrated testing, because by zooming in on the class-level, you don’t know if the controller will behave well when the application is actually used. Is the routing configuration correct? Can the framework resolve all of the controller’s arguments? Will dependencies be injected properly? And so on.

The alternative I mentioned in 2012 is to write functional tests for your controller. But this is not preferable in the end. These tests are slow and fragile, because you end up invoking much more code than just the domain logic.

As a rather unusual pastime for the Saturday night I attended the third Domain-Driven Design Africa online meetup. Thomas Pierrain a.k.a. use case driven spoke about his adaptation of Hexagonal architecture. “It’s not by the book,” as he said, but it solves a lot of the issues he encountered over the years. I’ll try to summarize his approach here, but I recommend watching the full talk as well.

Hexagonal architecture

Hexagonal architecture makes a distinction between the use cases of an application, and how they are connected to their surrounding infrastructure. Domain logic is represented by pure code (in the FP sense of the word), surrounded by a set of adapters that expose the use cases of the application to actual users and connect the application to databases, messages queues, and so on.

100% done

I’m happy to announce that my latest book “Advanced Web Application Architecture” is now complete. With ~390 pages it has become a well-rounded book full of useful design patterns and architectural principles built around the notion of object-pure code, that will help you create decoupled applications that are testable by definition, and support a domain-first approach to software development.

Use this link to get 10% off the price: https://leanpub.com/web-application-architecture/c/RELEASE_DAY

In this series, we’ve discussed several topics already. We talked about persistence and time, the filesystem and randomness. The conclusion for all these areas: whenever you want to “mock” these things, you may look for a solution at the level of programming tools used (use database or filesystem abstraction library, replace built-in PHP functions, etc.). But the better solution is always: add your own abstraction. Start by introducing your own interface (including custom return types), which describes exactly what you need. Then mock this interface freely in your application. But also provide an implementation for it, which uses “the real thing”, and write an integration test for just this class.

In a previous article, we discussed “persistence” and “time” as boundary concepts that need mocking by means of dependency inversion: define your own interface, then provide an implementation for it. There were three other topics left to cover: the filesystem, the network and randomness.

Mocking the filesystem

We already covered “persistence”, but only in the sense that we sometimes need a way to make in-memory objects persistent. After a restart of the application we should be able to bring back those objects and continue to use them as if nothing happened.

More and more I’ve come to realize that I’ve been mocking less and less.

The thing is, creating test doubles is a very dangerous activity. For example, what I often see is something like this:

$entityManager = $this->createMock(EntityManager::class);

$entityManager->expects($this->once())
    ->method('persist')
    ->with($object);
$entityManager->expects($this->once())
    ->method('flush')
    ->with($object);

Or, what appears to be better, since we’d be mocking an interface instead of a concrete class:

$entityManager = $this->createMock(ObjectManagerInterface::class);
// ...

To be very honest, there isn’t a big different between these two examples. If this code is in, for example, a unit test for a repository class, we’re not testing many of the aspects of the code that should have been tested instead.

In the previous article we discussed a sensible layer system, consisting of three layers:

• Domain
• Application
• Infrastructure

Infrastructure

The infrastructure layer, containing everything that connects the application’s use cases to “the world outside” (like users, hardware, other applications), can become quite large. As I already remarked, a lot of our software consists of infrastructure code, since that’s the realm of things complicated and prone to break. Infrastructure code connects our precious clean code to:

The first key concept of what I think is a very simple, at the very least “clean” architecture, is the concept of a layer. A layer itself is actually nothing, if you think about it. It’s simply determined by how it’s used. Let’s stay a bit philosophical, before we dive into some concrete architectural advice.

Qualities of layers

A layer in software serves the following (more or less abstract) purposes:

Looking back at my old blog posts, I think it’s good to write down a more balanced view on application architecture than the one that speaks from some of the older posts from 2013 and 2014. Before I do, I allow myself a quick self-centered trip down memory lane.

Why Symfony? Seven facts

The archive tells an interesting story about how my thoughts about software development changed over time. It all started with my first post in October 2011, How to make your service use tags. Back in those days, version 2 of the Symfony framework was still quite young. I had been working with symfony (version 1, small caps) for a couple of years and all of a sudden I was using Symfony2, and I absolutely loved this new shiny framework. I learned as much as I could about it (just as I had done with version 1), and eventually I became a Certified Symfony Developer, at the first round of exams held in Paris, during a Symfony Live conference. During those years I blogged and spoke a lot about Symfony, contributed to the documentation, and produced several open source bundles. I also wrote a book about it: A Year With Symfony.

It was quite a ride so far. We have seen commands, command buses, events and event buses. We distilled some more knowledge about them while formulating answers to some interesting questions from readers.

Why you should not dispatch events while handling a command

In a previous post we discussed a sample event (the UserSignedUp event):

class UserSignedUp implements Event
{
    public function name()
    {
        return 'user_signed_up';
    }

    public function __construct($userId)
    {
        $this->userId = $userId;
    }

    public function userId()
    {
        return $this->userId;
    }
}

An instance of such an event can be handed over to the event bus. It will look for any number of event handlers that wants to be notified about the event. In the case of the UserSignedUp event, one of the interested event handlers is the SendWelcomeMailWhenUserSignedUp handler:

So far we’ve had three posts in this series about commands, events and their corresponding buses and handlers:

• A wave of command buses
• Responsibilities of the command bus
• From commands to events

Now I’d like to take the time to answer some of the very interesting questions that by readers.

The difference between commands and events

Robert asked:

[…], could you possibly explain what are the main differences between a command bus and an even dispatcher?

In the previous posts we looked at commands and the command bus. Commands are simple objects which express a user’s intention to change something. Internally, the command object is handed over to the command bus, which performs the change that has been requested. While it eventually delegates this task to a dedicated command handler, it also takes care of several other things, like wrapping the command execution in a database transaction and protecting the original order of commands.

In the previous post we looked at commands and how you can use them to separate technical aspects of the input, from the actual behavior of your application. Commands are simple objects, handed over to the command bus, which performs the change that is needed.

As we learned, the command bus eventually calls the command handler which corresponds to the given command object. For example when a SignUp command is provided, the SignUpHandler will be asked to handle the command. So the command bus contains some kind of a lookup mechanism to match commands with their handlers. Some command bus libraries use a naming convention here (e.g. handler name = command name + “Handler”), some use a kind of service locator, etc.

Recently many people in the PHP community have been discussing a thing called the “command bus”. The Laravel framework nowadays contains an implementation of a command bus and people have been talking about it in several vodcasts.

My interest was sparked too. Last year I experimented with LiteCQRS but in the end I developed a collection of PHP packages known as SimpleBus which supports the use of commands and events in any kind of PHP application (there is a Symfony bridge too, if you like that framework). I also cover the subject of commands, events and their corresponding buses extensively during my Hexagonal Architecture workshop.

An ever recurring pattern in my life is this one:

1. I stumble upon some interesting piece of code, an intriguing book chapter, a fascinating concept, etc.
2. Slowly, over the next couple of weeks, my brain realises that, yes, this is some very interesting stuff.
3. Then I want to become all productive about it - writing things on my blog, speaking about it in public, maybe even writing a book about it.

This time it was domain-driven design (DDD), command-query responsibility segregation (CQRS) and in particular its architectural and technical aspects. While playing with existing libraries I soon recognized the huge benefits of applying hexagonal architecture and some of the tactical DDD patterns to a (Symfony) codebase.

This article shows an example of framework decoupling. You’ll find a more elaborate discussion in my latest book, Recipes for Decoupling.

Why would it be nice to decouple your user model from the framework’s security user or authentication model?

Reason 1: Hexagonal architecture

I like to use hexagonal architecture in my applications, which means among other things that the entities from my domain model stay behind a port. They are never exposed to, for instance, a controller, or a template. Whenever I want to show anything to the user, I create a dedicated view model for it.

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

Years ago my co-worker Maurits introduced me to the term “magic” in programming. He also provided the valuable dichotomy of convention and configuration (or in fact, he’d choose configuration over convention…). I think this distinction could be very helpful in psychological research, figuring out why some people prefer framework X over framework Y. One requires the developer to spell out everything they want in elaborate configuration files, the other relies on convention: placing certain files with certain names and certain methods in certain places will make everything work “magically”.

Paradigm; a nice word that means “a theory or a group of ideas about how something should be done, made, or thought about” (Merriam-Webster). In software development we have them too. From the philosophy and history of science courses I’ve followed, I remember that scientists working with different paradigms have great difficulty understanding each other, and appreciating each other’s work.

Paradigm Shifts

An example of a paradigm is the theory that the sun revolves around the earth. To a certain extent this is a fruitful theory, and it has been used for thousands of years. There’s of course another paradigm: the theory that the earth revolves around the sun. This is also a fruitful theory, and it can be used to explain a lot of observations, more than the previous theory. Still, people got angry with each other for moving the earth out of the center of the universe. Paradigm changes, or shifts, occur when the old theory has been stretched too much. It becomes impossible to hold on to it. Then some people start to experiment with a completely different paradigm, one that sounds totally weird, but in the end proves to have more power.

This is just a quick post sharing something I was able to figure out after doing some research.

The situation: our application throws exceptions by means of “talking back to the user”. As developer we don’t want to be notified about all these exceptions. They aren’t as important as any other exception that should be considered “critical”. Still, we do want to find these exceptions in the logs, because they can sometimes provide valuable feedback about the usability of the system.

When Symfony2 was created I first learned about the functional test, which is an interesting type of test where everything about your application is as real as possible. Just like with an integration or end-to-end test. One big difference: the test runner exercises the application’s front controller programmatically, instead of through a web server. This means that the input for the test is a Request object, not an actual HTTP message.

So, here’s a bit of a personal blog post for once.

Symfony trademark policy

I saw this tweet:

Suite à une mise en demeure, j’ai retiré les tutoriels qui concernent Symfony du Site. Il n’y aura pas de futures vidéos sur le Framework.

— Grafikart (@grafikart_fr) March 11, 2017

Meaning: “Following a formal notice, I removed the tutorials that are related to Symfony from the Site. There will be no future videos on the Framework.”

About 2 years ago I created a package that combines the power of two famous Symfony components: the Form component and the Console component. In short: this package allows you to interactively fill in a form by typing in the answers at the CLI. When I started working on it, this seemed like a pretty far-fetched idea. However, it made a lot of sense to me in terms of a ports & adapters architecture that I was looking for back then (and still always am, by the way). We could (and often should) write the code in our application layer in such a way that it doesn’t make a big difference whether we call applications services from a web controller or from a CLI “controller”.

This article shows an example of framework decoupling. You’ll find a more elaborate discussion in my latest book, Recipes for Decoupling.

Why would it be nice to decouple your user model from the framework’s security user or authentication model?

Reason 1: Hexagonal architecture

I like to use hexagonal architecture in my applications, which means among other things that the entities from my domain model stay behind a port. They are never exposed to, for instance, a controller, or a template. Whenever I want to show anything to the user, I create a dedicated view model for it.

Please note: After writing this article I’ve turned it into an official Cookbook article. After some time it has been unrecognizably rewritten by someone else.

Symfony2 firewalls depend for their authentication on UserProviders. These providers are requested by the authentication layer to provide a User object, for a given username. Symfony will check whether the password of this User is correct (i.e. verify it’s password) and will then generate a security token, so the user may stay authenticated during the current session. Out of the box, Symfony has a “in_memory” user provider and an “entity” user provider. In this post I’ll show you how to create your own UserProvider. The UserProvider in this example, tries to load a Yaml file containing information about users in the following format:

This article is about a topic that didn’t make the cut for my latest book, Recipes for Decoupling. It still contains some useful ideas I think, so here it is anyway!

DateTimeImmutable is mutable

I don’t know where I first heard it, but PHP’s DateTimeImmutable is not immutable:

<?php

$dt = new DateTimeImmutable('now');
echo $dt->getTimestamp() . "\n";

$dt->__construct('tomorrow');
echo $dt->getTimestamp() . "\n";

The result:

1656927919
1656972000

Indeed, DateTimeImmutable is not really immutable because its internal state can be modified after instantiation. After calling __construct() again, any existing object reference will have the modified state as well. But an even bigger surprise might be that if a constructor is public, it’s just another method that you can call. You’re not supposed to, but you can. Which is another reason to make the constructor of non-service objects private and add a public static constructor to a class that you want to be immutable:

As was the case with the SymfonyConsoleForm library, I realized I had not taken the time to properly introduce another library I created some time ago: ConvenientImmutability. This one is a lot simpler, but from a technical standpoint I think it might be interesting to at least talk a bit about the considerations that went into it.

For starters, it’s important to keep in mind that I don’t think you’ll actually need to use this library. Of course, I’ll explain why. But first: what is ConvenientImmutability about?

This article is about a topic that didn’t make the cut for my latest book, Recipes for Decoupling. It still contains some useful ideas I think, so here it is anyway!

DateTimeImmutable is mutable

I don’t know where I first heard it, but PHP’s DateTimeImmutable is not immutable:

<?php

$dt = new DateTimeImmutable('now');
echo $dt->getTimestamp() . "\n";

$dt->__construct('tomorrow');
echo $dt->getTimestamp() . "\n";

The result:

1656927919
1656972000

Indeed, DateTimeImmutable is not really immutable because its internal state can be modified after instantiation. After calling __construct() again, any existing object reference will have the modified state as well. But an even bigger surprise might be that if a constructor is public, it’s just another method that you can call. You’re not supposed to, but you can. Which is another reason to make the constructor of non-service objects private and add a public static constructor to a class that you want to be immutable:

In essence, everything is a string.

Well, you can always go one layer deeper and find out what a string really is, but for web apps I work on, both input data and output data are strings. The input is an HTTP request, which is a plain-text message that gets passed to the web server, the PHP server, the framework, and finally a user-land controller. The output is an HTTP response, which is also a plain-text message that gets passed to the client. If my app needs the database to load or store some data, that data too is in its initial form a string. It needs to be deserialized into objects to do something and later be serialized into strings so we can store the results.

In “Negative Architecture”, Michael Feathers speaks about certain aspects of software architecture leading to some kind of negative architecture. Feathers mentions the IO Monad from Haskell (functional programming) as an example, but there are parallel examples in object-oriented programming. For example, by using layers and the dependency inversion principle you can “guarantee” that a certain class, in a certain layer (e.g. domain, application) won’t do any IO - no talking to a database, no HTTP requests to some remote service, etc.

Consider the following rule:

When you create an object, it should be complete, consistent and valid in one go.

It is derived from the more general principle that it should not be possible for an object to exist in an inconsistent state. I think this is a very important rule, one that will gradually lead everyone from the swamps of those dreaded “anemic” domain models. However, the question still remains: what does all of this mean?

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

My new book Recipes for Decoupling is 100% complete and available now!

And now some other news related to this book.

A little background information

My new book is based on two things: 20 years of experience with (mostly framework) coupling issues in legacy code, and the hypothesis that PHPStan, the automated static analysis tool for PHP, can help us keep our code decoupled. Decoupling often means we want to use a dependency, but don’t want to couple our code too tightly to it. The process of decoupling often involves some kind of rule, like “don’t use Container::get()”, but we don’t want to focus on this rule all the time, or explain it to new members of the team. Instead, we want a tool that shows you an error when you don’t follow the rule. Such an error will break the build of the project and prevent you from ever coupling to a dependency in this specific way again.

I was tweeting something about having separate “DDD” and “ORM” entities in a project in a project, and that I don’t understand this. There were some great comments and questions, thanks a lot for that! To be honest, I understand more about it now. In this article I’ll try to provide some more information about this.

I’m glad many developers want to use input from the book “Domain-Driven Design” by Eric Evans to improve their domain model. I recommend reading this book and getting your information from the source, because unfortunately the internet, tweets, e-books, including my own books, aren’t able to reflect a full, nor a correct view of everything there is to find out about this topic. All too often DDD is completely misinterpreted to be “an elitist, dogmatic approach to programming, where we use DTOs, layers, and CQRS”…

Years ago my co-worker Maurits introduced me to the term “magic” in programming. He also provided the valuable dichotomy of convention and configuration (or in fact, he’d choose configuration over convention…). I think this distinction could be very helpful in psychological research, figuring out why some people prefer framework X over framework Y. One requires the developer to spell out everything they want in elaborate configuration files, the other relies on convention: placing certain files with certain names and certain methods in certain places will make everything work “magically”.

Paradigm; a nice word that means “a theory or a group of ideas about how something should be done, made, or thought about” (Merriam-Webster). In software development we have them too. From the philosophy and history of science courses I’ve followed, I remember that scientists working with different paradigms have great difficulty understanding each other, and appreciating each other’s work.

Paradigm Shifts

An example of a paradigm is the theory that the sun revolves around the earth. To a certain extent this is a fruitful theory, and it has been used for thousands of years. There’s of course another paradigm: the theory that the earth revolves around the sun. This is also a fruitful theory, and it can be used to explain a lot of observations, more than the previous theory. Still, people got angry with each other for moving the earth out of the center of the universe. Paradigm changes, or shifts, occur when the old theory has been stretched too much. It becomes impossible to hold on to it. Then some people start to experiment with a completely different paradigm, one that sounds totally weird, but in the end proves to have more power.

With excerpts from the documentation

I have worked with symfony 1, Symfony 2+, Zend Framework 1, Zend Expressive, but never with Laravel. Until now. My partner was looking for a way to learn PHP and building web applications with it. Most of my own framework knowledge is related to Symfony, so my initial plan was to find a Symfony course for her. However, Jeffrey Way’s Laracasts also came to mind, and I thought it would be an interesting learning experience for us both if Laravel would be the framework of choice in this matter. It turned out to be a good idea. My partner is making good progress, and I get to see what Laravel is all about. We have a lot of fun together, finding out how it works, or what you’re supposed to be doing with it.

Last year I took several Scala courses on Coursera. It was an interesting experience and it has brought me a lot of new ideas. One of these is the idea of a singleton object (as opposed to a class). It has the following characteristics:

• There is only one instance of it (hence it’s called a “singleton”, but isn’t an implementation of the Singleton design pattern).
• It doesn’t need to be explicitly instantiated (it doesn’t have the traditional static getInstance() method). In fact, an instance already exists when you first want to use it.
• There doesn’t have to be built-in protection against multiple instantiations (as there is and can only be one instance, by definition).

Converting this notion to PHP is impossible, but if it was possible, you could do something like this:

Recently many people in the PHP community have been discussing a thing called the “command bus”. The Laravel framework nowadays contains an implementation of a command bus and people have been talking about it in several vodcasts.

My interest was sparked too. Last year I experimented with LiteCQRS but in the end I developed a collection of PHP packages known as SimpleBus which supports the use of commands and events in any kind of PHP application (there is a Symfony bridge too, if you like that framework). I also cover the subject of commands, events and their corresponding buses extensively during my Hexagonal Architecture workshop.

I was tweeting something about having separate “DDD” and “ORM” entities in a project in a project, and that I don’t understand this. There were some great comments and questions, thanks a lot for that! To be honest, I understand more about it now. In this article I’ll try to provide some more information about this.

I’m glad many developers want to use input from the book “Domain-Driven Design” by Eric Evans to improve their domain model. I recommend reading this book and getting your information from the source, because unfortunately the internet, tweets, e-books, including my own books, aren’t able to reflect a full, nor a correct view of everything there is to find out about this topic. All too often DDD is completely misinterpreted to be “an elitist, dogmatic approach to programming, where we use DTOs, layers, and CQRS”…

I was tweeting something about having separate “DDD” and “ORM” entities in a project in a project, and that I don’t understand this. There were some great comments and questions, thanks a lot for that! To be honest, I understand more about it now. In this article I’ll try to provide some more information about this.

I’m glad many developers want to use input from the book “Domain-Driven Design” by Eric Evans to improve their domain model. I recommend reading this book and getting your information from the source, because unfortunately the internet, tweets, e-books, including my own books, aren’t able to reflect a full, nor a correct view of everything there is to find out about this topic. All too often DDD is completely misinterpreted to be “an elitist, dogmatic approach to programming, where we use DTOs, layers, and CQRS”…

I’d like to start this article with a quote from Ross Tuck’s article “Persisting Value Objects in Doctrine”. He describes different ways of persisting value objects when using Doctrine ORM. At the end of the page he gives us the following option - the “nuclear” one:

[…] Doctrine is great for the vast majority of applications but if you’ve got edge cases that are making your entity code messy, don’t be afraid to toss Doctrine out. Setup an interface for your repositories and create an alternate implementation where you do the querying or mapping by hand. It might be a PITA but it might also be less frustration in the long run.

It appears that I didn’t make myself clear while writing about entity managers and manager registries yesterday. People were quick to reply that instead you should inject entity repositories. However, I wasn’t talking about entity repositories here. I was talking about classes that get an EntityManager injected because they want to call persist() or flush(). The point of my previous post was that in those cases you should inject the manager registry, because you don’t know beforehand which entity manager manages the entities you are trying to persist. By injecting a manager registry you also make your code useful in contexts where another Doctrine persistence library is used.

Please also read my next article about injecting entity managers.

Or: how to make your persistence-related code more reusable

You probably know that you should trim your controllers as much as possible. This usually means extracting some classes from the semi-procedural code inside your controller class. You create services for the extracted classes and inject the necessary dependencies into them. You can no longer use the convenient $this->getDoctrine()->getManager(). Instead you inject the doctrine.orm.default_entity_manager service into every service that needs the EntityManager to persist data in your relational database:

The one simple trick to do a better job as a programmer is to git commit as if your commit could be accidentally deployed (and it wouldn’t break the production environment…)

Why would this improve your work? Because it pushes for improvements in several areas.

If a commit needs to leave the project in a working state, you need to:

1. Be able to prove that after making your changes, the project is in a working state. This forces you to put automated tests in place.
2. Be able to make changes that don’t break existing code, or that only add new features. This forces you think about first improving the existing code design to allow for your change to happen.

To practice with the latter, I recommend learning about the Mikado method. It makes it easier to recognize prerequisites for the change you want to make, and then forces you to enable the real goals by implementing the prerequisites first.

Paradigm; a nice word that means “a theory or a group of ideas about how something should be done, made, or thought about” (Merriam-Webster). In software development we have them too. From the philosophy and history of science courses I’ve followed, I remember that scientists working with different paradigms have great difficulty understanding each other, and appreciating each other’s work.

Paradigm Shifts

An example of a paradigm is the theory that the sun revolves around the earth. To a certain extent this is a fruitful theory, and it has been used for thousands of years. There’s of course another paradigm: the theory that the earth revolves around the sun. This is also a fruitful theory, and it can be used to explain a lot of observations, more than the previous theory. Still, people got angry with each other for moving the earth out of the center of the universe. Paradigm changes, or shifts, occur when the old theory has been stretched too much. It becomes impossible to hold on to it. Then some people start to experiment with a completely different paradigm, one that sounds totally weird, but in the end proves to have more power.

You say "convention over configuration;" I hear "ambient information stuck in someone's head." You say "configuration over hardcoding;" I hear "information in a different language that must be parsed, can be malformed, or not exist."

— Paul Snively (@paul_snively) March 2, 2019

Dependency injection is very important. Dependency injection containers are too. The trouble is with the tools, that let us define services in a meta-language, and rely on conventions to work well. This extra layer requires the “ambient information” Paul speaks about in his tweet, and easily lets us make mistakes that we wouldn’t make if we’d just write out the code for instantiating our services.

Statically fetching dependencies

I’ve worked with several code bases that were littered with calls to Zend_Registry::get(), sfContext::getInstance(), etc. to fetch a dependency when needed. I’m a little afraid to mention façades here, but they also belong in this list. The point of this article is not to bash a certain framework (they are all lovely), but to show how to get rid of these “centralized dependency managers” when you need to. The characteristics of these things are:

The problem

With a new bundle release you may want to rename services or parameters, make a service private, change some constructor arguments, change the structure of the bundle configuration, etc. Some of these changes may acually be backwards incompatible changes for the users of that bundle. Luckily, the Symfony DependenyInjection component and Config component both provide you with some options to prevent such backwards compatibility (BC) breaks. If you want to know more about backwards compatibility and bundle versioning, please read my previous article on this subject.

As you may have read on the Symfony blog: as of Symfony 2.4 you can register console commands using the service tag console.command.

What is the big deal, you would say. Well, let’s discover it now. This is how the documentation tells you to write your console commands:

namespace Matthias\ConsoleBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class MyCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this->setName('my:action');
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        // do something
    }
}

How to make sure your command will be noticed?

Until Symfony 2.4 there were two requirements for your command to be picked up by the console application itself:

Unit tests are tests on a micro-scale. When unit testing, you are testing little units of your code to make sure that, given a certain input, they produce the output you expected. When your unit of code makes calls to other objects, you can “mock” or “stub” these objects and verify that a method is called a specific number of times, or to make sure the unit of code you’re testing will receive the correct data from the other objects.

Below you will find the slides for my presentation about Dependency Injection Smells (see also one of my previous posts from which this idea originated). If you have any questions, or feedback, please let me know by posting a comment below (or on joind.in).

Dependency Injection Smells from matthiasnoback

This week is Dutch PHP Conference week! It is something I’m very excited about. Three years ago I attended this conference for the first time. It was also my first time to visit any (PHP) conference at all, and this is a big conference, with lots of speakers, lots of attendees and multiple tracks. The Dutch PHP Conference (DPC) has a certain grandeur - many large rooms, many famous developers speaking. It is an inspiring event. Visiting the conference back in 2010 lit a fire in me to start investigating anything there was to know about writing good code: reusable, clean code, with well named, short and testable methods, readable and maintainable by others. In other words, I started being critical of the code I produced.

The Symfony2 DependencyInjection Component has made my life and work as a developer a lot easier. Choosing the right way to use it however can be a bit difficult sometimes. Knowing what a/the service container can do, helps a lot, and also thinking about how you would do it with just PHP can put you back on track. To be able to recognize some problems related to dependency injection in your own code, I will describe a few “dependency injection smells” below (a term derived from “code smells”, used by Kent Beck, Martin Fowler and the likes).

The class Silex\Application extends \Pimple, a very lean dependency injection container, which itself implements \ArrayAccess. This allows you to define parameters and services and retrieve them from the Application like keys in an array. The first time I defined a service in a Silex application, in my case the Buzz Browser, I did it “inline”, i.e. inside my app.php file. This is how I did it:

use Buzz\Browser;

$app = new Silex\Application();

$app['autoloader']->registerNamespace('Buzz', __DIR__.'/vendor/buzz/lib');

$app['buzz.client_class'] = 'Buzz\\Client\\Curl';

$app['browser'] = $app->share(function() use ($app) {
    $clientClass = $app['buzz.client_class'];
    return new Browser(new $clientClass);
});

// $app['browser'] is ready for use

But this is not very reusable; every Silex application that needs Buzz\Browser as a service, needs to take care of the autoloading, configure and define the service. The Browser service is not such a very complex service, but think about everything that needs to be done to define and configure a service like the Doctrine DBAL (of course, Silex has a DoctrineServiceProvider for that already…).

Earlier I was looking for an alternative to Symfony1’s routing.load_configuration event, so I could add some extra routes on-the-fly. This may be useful, when routes change in more ways than only variable request parameters as part of routes do (you know, like /blog/{id}). I got it completely wrong in my previous post about this subject. Of course, adding extra routes is a matter of creating a custom routing loader, and tell the framework about it using the service container. So, there we go.

Please note: There is also an interesting Cookbook article about the same subject (not written by me).

With Symfony2, many things are managed through dependency injection. Except for forms. Oh, wait, forms can be services too of course! Remember? Any class instance can be a service… Now, as in many other cases, Symfony2’s FrameworkBundle adds some magic to the Form component by creating services that link several parts of the Form component together. For example, depending on the value of the framework.csrf_protection parameter in config.yml file a hidden field “_token” will be added to all forms, site-wide. To create this kind of very general “form extension” in your own project, you need to do just a few things (I am going to use the example of a “Captcha” form extension, but I don’t give a full implementation of this idea).

One of the ugly things about Symfony 1 validators was that their dependencies were generally fetched from far away, mainly by calling sfContext::getInstance(). With Symfony2 and it’s service container, this isn’t necessary anymore. Validators can be services. The example below is quite simple, but given you can inject anything you want, you can make it quite complicated. I show you how to create a validator that checks with the router of the value-under-validation is a route.

Please note: After writing this article I’ve started to write more documentation for the Config component. First I published it here, then it became the official documentation of the Config component.

After reading the Cookbook article How to expose a Semantic Configuration for a Bundle I became interested in the possibilities for defining a configuration tree using the TreeBuilder from the Symfony component Config. This TreeBuilderallows you to build a schema against which the configuration values (or short: config) from/app/config.yml` and the like are to be validated.

Sometimes you need to make changes to the Response object, after it is returned by your controller, but before it is rendered as output to the client (e.g. the browser). You may want to set some extra response headers, or “completely mess up the content” of the response. You can accomplish this by creating an event listener that listens to the kernel.response event. I give you a sample event listener which changes the Content-Type header in case the requested format is “json” and the browser’s accepted response format contains “text/html”; in that case, at least in my experience, the browser doesn’t render the JSON string as plain text when the status code is 4xx or 5xx. So in these situations, the event listener changes the “Content-Type” to “text/plain”, to be sure you always get decent output in the browser.

Please note: After writing this article I’ve modified it to become part of the official Symfony documentation as a Cookbook article. Afterwards it has been moved to the Dependency Injection Component documentation.

First of all: dependency injection is GREAT!

Several of Symfony2’s core services depend on tags to recognize which user-defined services should be loaded, notified of events, etc. For example, Twig uses the tag twig.extension to load extra extensions. It would also be great to use tags in case your service implements some kind of “chain”, in which several alternative strategies are tried until one of them is successful. In this post I use the example of a so-called “TransportChain”. This chain consists of a set of classes which implement the Swift_Transport interface. Using the chain, the mailer may try several ways of transport, until one succeeds. This post focuses on the “dependency injection” part of the story. The implementation of the real TransportChain is left to the reader (as an exercise ;)).

I’ve mentioned this several times without explaining: the rule that every class should have a test, or that every class method should have a test, does not make sense at all. Still, it’s a rule that many teams follow. Why? Maybe they used to have a #NoTest culture and they never want to go back to it, so they establish a rule that is easy to enforce. When reviewing you only have to check: does the class have a test? Okay, great. It’s a bad test? No problem, it is a test. I already explained why I think you need to make an effort to not just write any test, but to write good tests (see also: Testing Anything; Better Than Testing Nothing?). In this article I’d like to look closer at the numbers: one class - one test.

I’m a big fan of the BDD Books by Gáspár Nagy and Seb Rose, and I’ve read a lot about writing and improving scenarios, like Specification by Example by Gojko Adzic and Writing Great Specifications by Kamil Nicieja. I can recommend reading anything from Liz Keogh as well. Trying to apply their suggestions in my development work, I realized: specifications benefit from good writing. Writing benefits from good thinking. And so does design. Better writing, thinking, designing: this will make us do a better job at programming. Any effort put into these activities has a positive impact on the other areas, even on the code itself.

“Yes, I know. Our tests aren’t perfect, but it’s better to test anything than to test nothing at all, right?”

Let’s look into that for a bit. We’ll try the “Fowler Heuristic” first:

One of my favourite (of the many) things I learned from consulting with Martin Fowler is that he would often ask “Compared to what?”

• Agile helps you ship faster!
• Compared to what?

[…]

Often there is no baseline.

Whenever I read a test method I want to understand it without having to jump around in the test class (or worse, in dependencies). If I want to know more, I should be able to “click” on one of the method calls and find out more.

I’ll explain later why I want this, but first I’ll show you how to get to this point.

As an example, here is a test I encountered recently:

@ediar asked me on Twitter if I still think a constructor should not be tested. It depends on the type of object you’re working with, so I think it’ll be useful to elaborate here.

Would you test the constructor of a service that just gets some dependencies injected? No. You’ll test the behavior of the service by calling one of its public methods. The injected dependencies are collaborating services and the service as a whole won’t work if anything went wrong in the constructor.

I recently heard this interesting question: if your project uses a static analysis tool like PHPStan or Psalm (as it should), should the tests by analysed too?

The first thing to consider: what are potential reasons for not analysing your test code?

Why not?

1. Tests are messy in terms of types

Tests may use mocks, which can be confusing for the static analyser:

$object = $this->createMock(MockedInterface::class);

The actual type of $object is an intersection type, i.e. $object is both a MockObject and a MockedInterface, but the analyser only recognizes MockObject. You may not like all those warnings about “unknown method” calls on MockObject $object so you exclude test code from the analysis.

A lot can happen in 9 years. Back then I was still advocating that you should unit-test your controllers and that setter injection is very helpful when replacing controller dependencies with test doubles. I’ve changed my mind: constructor injection is the right way for any service object, including controllers. And controllers shouldn’t be unit tested, because:

• Those unit tests tend to be a one-to-one copy of the controller code itself. There is no healthy distance between the test and the implementation.
• Controllers need some form of integrated testing, because by zooming in on the class-level, you don’t know if the controller will behave well when the application is actually used. Is the routing configuration correct? Can the framework resolve all of the controller’s arguments? Will dependencies be injected properly? And so on.

The alternative I mentioned in 2012 is to write functional tests for your controller. But this is not preferable in the end. These tests are slow and fragile, because you end up invoking much more code than just the domain logic.

As a rather unusual pastime for the Saturday night I attended the third Domain-Driven Design Africa online meetup. Thomas Pierrain a.k.a. use case driven spoke about his adaptation of Hexagonal architecture. “It’s not by the book,” as he said, but it solves a lot of the issues he encountered over the years. I’ll try to summarize his approach here, but I recommend watching the full talk as well.

Hexagonal architecture

Hexagonal architecture makes a distinction between the use cases of an application, and how they are connected to their surrounding infrastructure. Domain logic is represented by pure code (in the FP sense of the word), surrounded by a set of adapters that expose the use cases of the application to actual users and connect the application to databases, messages queues, and so on.

Recently I received a question; if I could explain these four lines:

/**
 * @test
 */
public function it_works_with_a_standard_use_case_for_command_objects(): void

The author of the email had some great points.

1. For each, my test I should write +3 new line of code instead write, public function testItWorksWithAStandardUseCaseForCommandObjects(): void

2. PSR-12 say that “Method names MUST be declared in camelCase”. [The source of this is actually PSR-1].

3. In PHPUnit documentation author say “The tests are public methods that are named test*” and left example below

When Symfony2 was created I first learned about the functional test, which is an interesting type of test where everything about your application is as real as possible. Just like with an integration or end-to-end test. One big difference: the test runner exercises the application’s front controller programmatically, instead of through a web server. This means that the input for the test is a Request object, not an actual HTTP message.

During a recent run of my Advanced Web Application Architecture training, we discussed the distinction between infrastructure code and non-infrastructure code, which I usually call core code. One of the participants summarized the difference between the two as: “everything in your vendor directory is infrastructure code”. I don’t agree with that, and I will explain why in this article.

Not all code in vendor is infrastructure code

Admittedly, it’s easy for anyone to not agree with a statement like this, because you can simply make up your own definitions of “infrastructure” that turn the statement false. As a matter of fact, I’m currently working on my next book (which has the same title as the training), and I’m working on a memorable definition that covers all the cases. I’ll share with you the current version of that definition, which consists of two rules defining core code. Any code that doesn’t follow both these rules at the same time, should be considered infrastructure code.

Triggered by Marco Pivetta who apparently said during his talk at Symfony Live Berlin: “If you still don’t have tests, this is unprofessional”, I thought I’d tweet about that too: “It’s good for someone to point this out from time to time”.

I don’t like it when a blog post is about tweets, just as I don’t like it when a news organization quotes tweets; as if they are some important source of wisdom. But since this kind of tweet tends to invoke many reactions (and often emotionally charged ones too), I thought it would be smart to get the discussion off Twitter and write something more nuanced instead.

In part 1 of this short series (it’s going to end with this article) we covered how you can test-drive the queries in a repository class. Returning query results is only part of the job of a repository though. The other part is to store objects (entities), retrieve them using something like a save() and a getById() method, and possibly delete them. Some people will implement these two jobs in one repository class, some like to use two or even many repositories for this. When you have a separate write and read model (CQRS), the read model repositories will have the querying functionality (e.g. find me all the active products), the write model repositories will have the store/retrieve/delete functionality.

There’s something I’ve only been doing since a year or so, and I’ve come to like it a lot. My previous testing experiences were mostly at the level of unit tests or functional/system tests. What was left out of the equation was integration tests. The perfect example of which is a test that proves that your custom repository class works well with the type of database that you use for the project, and possibly the ORM or database abstraction library you use to talk with that database. A test for a repository can’t be a unit test; that wouldn’t make sense. You’d leave a lot of assumptions untested. So, no mocking is allowed.

System and integration tests need database fixtures. These fixtures should be representative and diverse enough to “fake” normal usage of the application, so that the tests using them will catch any issues that might occur once you deploy the application to the production environment. There are many different options for dealing with fixtures; let’s explore some of them.

Generate fixtures the natural way

The first option, which I assume not many people are choosing, is to start up the application at the beginning of a test, then navigate to specific pages, submit forms, click buttons, etc. until finally the database has been populated with the right data. At that point, the application is in a useful state and you can continue with the act/when and assert/then phases. (See the recent article “Pickled State” by Robert Martin on the topic of tests as specifications of a finite state machine).

The downsides of starting with the domain model

All the architectural focus on having a clean and infrastructure-free domain model is great. It’s awesome to be able to develop your domain model in complete isolation; just a bunch of unit tests helping you design the most beautiful objects. And all the “impure” stuff comes later (like the database, UI interactions, etc.).

However, there’s a big downside to starting with the domain model: it leads to inside-out development. The first negative effect of this is that when you start with designing your aggregates (entities and value objects), you will definitely need to revise them when you end up actually using them from the UI. Some aspects may turn out to be not so well-designed at all, and will make no sense from the user’s perspective. Some functionality may have been designed well, but only theoretically, since it will never actually be used by any real client, except for the unit test you wrote for it.

This article is part 2 of my review of the book “Fifty quick ideas to improve your tests”. I’ll continue to share some of my personal highlights with you.

Replace multiple steps with a higher-level step

If a test executes multiple tasks in sequence that form a higher-level action, often the language and the concepts used in the test explain the mechanics of test execution rather than the purpose of the test, and in this case the entire block can often be replaced with a single higher-level concept.

Review

After reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose, I picked up another book, which I bought a long time ago: “Fifty Quick Ideas to Improve Your Tests” by Gojko Adzic, David Evans, Tom Roden and Nikola Korac. Like with so many books, I find there’s often a “right” time for them. When I tried to read this book for the first time, I was totally not interested and soon stopped trying to read it. But ever since Julien Janvier asked me if I knew any good resources on how to write good acceptance test scenarios, I kept looking around for more valuable pointers, and so I revisited this book too. After all, one of the author’s of this book - Gojko Adzic - also wrote “Bridging the communication gap - Specification by example and agile acceptance testing”, which made a lasting impression on me. If I remember correctly, the latter doesn’t have too much practical advice on writing goods tests (or scenarios), and it was my hope that “Fifty quick ideas” would.

I’ve just finished reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose. It’s the first in a series of books about BDD (Behavior-Driven Development). The next parts are yet to be written/published. Part of the reason to pick up this book was that I’d seen it on Twitter (that alone would not be a sufficient reason of course). The biggest reason was that after delivering a testing and aggregate design workshop, I noticed that my acceptance test skills aren’t what they should be. After several years of not working as a developer on a project for a client, I realized again that (a quote from the book):

In this series, we’ve discussed several topics already. We talked about persistence and time, the filesystem and randomness. The conclusion for all these areas: whenever you want to “mock” these things, you may look for a solution at the level of programming tools used (use database or filesystem abstraction library, replace built-in PHP functions, etc.). But the better solution is always: add your own abstraction. Start by introducing your own interface (including custom return types), which describes exactly what you need. Then mock this interface freely in your application. But also provide an implementation for it, which uses “the real thing”, and write an integration test for just this class.

In a previous article, we discussed “persistence” and “time” as boundary concepts that need mocking by means of dependency inversion: define your own interface, then provide an implementation for it. There were three other topics left to cover: the filesystem, the network and randomness.

Mocking the filesystem

We already covered “persistence”, but only in the sense that we sometimes need a way to make in-memory objects persistent. After a restart of the application we should be able to bring back those objects and continue to use them as if nothing happened.

More and more I’ve come to realize that I’ve been mocking less and less.

The thing is, creating test doubles is a very dangerous activity. For example, what I often see is something like this:

$entityManager = $this->createMock(EntityManager::class);

$entityManager->expects($this->once())
    ->method('persist')
    ->with($object);
$entityManager->expects($this->once())
    ->method('flush')
    ->with($object);

Or, what appears to be better, since we’d be mocking an interface instead of a concrete class:

$entityManager = $this->createMock(ObjectManagerInterface::class);
// ...

To be very honest, there isn’t a big different between these two examples. If this code is in, for example, a unit test for a repository class, we’re not testing many of the aspects of the code that should have been tested instead.

Why code coverage for Behat?

PHPUnit has built-in several options for generating code coverage data and reports. Behat doesn’t. As Konstantin Kudryashov (@everzet) points out in an issue asking for code coverage options in Behat:

Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you’re doing code testing with StoryBDD - you’re doing it wrong.

He’s right I guess. The main issue is that StoryBDD isn’t about code, so it doesn’t make sense to calculate code coverage for it. Furthermore, the danger of collecting code coverage data and generating coverage reports is that people will start using it as a metric for code quality. And maybe they’ll even set management targets based on coverage percentage. Anyway, that’s not what this article is about…

Unbalanced test suites

Having spoken to many developers and development teams so far, I’ve recognized several patterns when it comes to software testing. For example:

1. When the developers use a framework that encourages or sometimes even forces them to marry their code to the framework code, they only write functional tests - with a real database, making (pseudo) HTTP requests, etc. using the test tools bundled with the framework (if you’re lucky, anyway). For these teams it’s often too hard to write proper unit tests. It takes too much time, or is too difficult to set up.
2. When the developers use a framework that enables or encourages them to write code that is decoupled from the framework, they have all these nice, often pretty abstract, units of code. Those units are easy to test. What often happens is that these teams end up writing only unit tests, and don’t supply any tests or “executable specifications” proving the correctness of the behavior of the application at large.
3. Almost nobody writes proper acceptance tests. That is, most tests that use a BDD framework, are still solely focusing on the technical aspects (verifying URLs, HTML elements, rows in a database, etc.), while to be truly beneficial they should be concerned about application behavior, defined in a ubiquitous language that is used by both the technical staff and the project’s stakeholders.

Please note that these are just some very rough conclusions, there’s nothing scientific about them. It would be interesting to do some actual research though. And probably someone has already done this.

In the introduction to this series I mentioned that testing object interactions can be really hard. Most unit testing tutorials cover this subject by introducing the PHPUnit mocking sub-framework. The word “mock” in the context of PHPUnit is given the meaning of the general concept of a “test double”. In reality, a mock is a very particular kind of test double. I can say after writing lots of unit tests for a couple of years now that my testing experience would have definitely been much better if I had known about the different kinds of test doubles that you can use in unit tests. Each type of test double has its own merits and it is vital to the quality of your test suite that you know when to use which one.

In the introduction article of this series I quickly mentioned that I think unit testing often focuses too much on assertions. The historic reason for this is that in introductory articles and workshops it is often said that:

1. You are supposed to pick the most specific assertion the testing framework offers.
2. You are supposed to have only one assertion in each test method.
3. You are supposed to write the assertion first, since that is the goal you are working towards.

I used to preach these things myself too (yes, “development with tests” often comes with a lot of preaching). But now I don’t follow these rules anymore. I will shortly explain my reasons. But before I do, let’s take a step back and consider something that is known as the Test framework in a tweet, by Mathias Verraes. It looks like this:

This is the introduction to a series of articles related to what I call: the “PHP testing experience”. I must say I’m not really happy with it. And so are many others I think. In the last couple of years I’ve met many developers who experienced a lot of trouble while trying to make testing a serious part of their development workflow. It is, I admit, a hard thing to accomplish. I see many people fail at it. Either the learning curve is too steep for them or they are lacking some insight into the concepts and reasoning behind testing. This has unfortunately led many of them to stop trying.

I’m a big fan of the BDD Books by Gáspár Nagy and Seb Rose, and I’ve read a lot about writing and improving scenarios, like Specification by Example by Gojko Adzic and Writing Great Specifications by Kamil Nicieja. I can recommend reading anything from Liz Keogh as well. Trying to apply their suggestions in my development work, I realized: specifications benefit from good writing. Writing benefits from good thinking. And so does design. Better writing, thinking, designing: this will make us do a better job at programming. Any effort put into these activities has a positive impact on the other areas, even on the code itself.

This article is part 2 of my review of the book “Fifty quick ideas to improve your tests”. I’ll continue to share some of my personal highlights with you.

Replace multiple steps with a higher-level step

If a test executes multiple tasks in sequence that form a higher-level action, often the language and the concepts used in the test explain the mechanics of test execution rather than the purpose of the test, and in this case the entire block can often be replaced with a single higher-level concept.

Review

After reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose, I picked up another book, which I bought a long time ago: “Fifty Quick Ideas to Improve Your Tests” by Gojko Adzic, David Evans, Tom Roden and Nikola Korac. Like with so many books, I find there’s often a “right” time for them. When I tried to read this book for the first time, I was totally not interested and soon stopped trying to read it. But ever since Julien Janvier asked me if I knew any good resources on how to write good acceptance test scenarios, I kept looking around for more valuable pointers, and so I revisited this book too. After all, one of the author’s of this book - Gojko Adzic - also wrote “Bridging the communication gap - Specification by example and agile acceptance testing”, which made a lasting impression on me. If I remember correctly, the latter doesn’t have too much practical advice on writing goods tests (or scenarios), and it was my hope that “Fifty quick ideas” would.

I’ve just finished reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose. It’s the first in a series of books about BDD (Behavior-Driven Development). The next parts are yet to be written/published. Part of the reason to pick up this book was that I’d seen it on Twitter (that alone would not be a sufficient reason of course). The biggest reason was that after delivering a testing and aggregate design workshop, I noticed that my acceptance test skills aren’t what they should be. After several years of not working as a developer on a project for a client, I realized again that (a quote from the book):

In essence, everything is a string.

Well, you can always go one layer deeper and find out what a string really is, but for web apps I work on, both input data and output data are strings. The input is an HTTP request, which is a plain-text message that gets passed to the web server, the PHP server, the framework, and finally a user-land controller. The output is an HTTP response, which is also a plain-text message that gets passed to the client. If my app needs the database to load or store some data, that data too is in its initial form a string. It needs to be deserialized into objects to do something and later be serialized into strings so we can store the results.

Whenever I read a test method I want to understand it without having to jump around in the test class (or worse, in dependencies). If I want to know more, I should be able to “click” on one of the method calls and find out more.

I’ll explain later why I want this, but first I’ll show you how to get to this point.

As an example, here is a test I encountered recently:

When Symfony2 was created I first learned about the functional test, which is an interesting type of test where everything about your application is as real as possible. Just like with an integration or end-to-end test. One big difference: the test runner exercises the application’s front controller programmatically, instead of through a web server. This means that the input for the test is a Request object, not an actual HTTP message.

There’s something I’ve only been doing since a year or so, and I’ve come to like it a lot. My previous testing experiences were mostly at the level of unit tests or functional/system tests. What was left out of the equation was integration tests. The perfect example of which is a test that proves that your custom repository class works well with the type of database that you use for the project, and possibly the ORM or database abstraction library you use to talk with that database. A test for a repository can’t be a unit test; that wouldn’t make sense. You’d leave a lot of assumptions untested. So, no mocking is allowed.

System and integration tests need database fixtures. These fixtures should be representative and diverse enough to “fake” normal usage of the application, so that the tests using them will catch any issues that might occur once you deploy the application to the production environment. There are many different options for dealing with fixtures; let’s explore some of them.

Generate fixtures the natural way

The first option, which I assume not many people are choosing, is to start up the application at the beginning of a test, then navigate to specific pages, submit forms, click buttons, etc. until finally the database has been populated with the right data. At that point, the application is in a useful state and you can continue with the act/when and assert/then phases. (See the recent article “Pickled State” by Robert Martin on the topic of tests as specifications of a finite state machine).

Several years ago, when the PHP-FIG (PHP Framework Interop Group) created its first PSRs (PHP Standard Recommendations) they started some big changes in the PHP ecosystem. The standard for class auto-loading was created to go hand-in-hand with the then new package manager Composer. PSRs for coding standards were defined, which I’m sure helped a lot of teams to leave coding standard discussions behind. The old tabs versus spaces debate was forever settled and the jokes about it now feel quite outdated.

I recently wrote about when to add an interface to a class. After explaining good reasons for adding an interface, I claim that if none of those reasons apply in your situation, you should just use a class and declare it “final”.

PHP 5 introduces the final keyword, which prevents child classes from overriding a method by prefixing the definition with final. If the class itself is being defined final then it cannot be extended.

Last week I wrote about when to add an interface to a class. The article finishes with the claim that classes from the application’s domain don’t usually need an interface. The reason is that domain code isn’t going to be swapped out with something else. This code is the result of careful modelling work that’s done based on the business domain that you work with. And even if you’d work on, say, two financial software projects in a row, you’ll find that the models you produce for each of them will be different in many subtle (if not radical) ways. Paradoxically you’ll find that in practice a domain model can sometimes be reused after all. There are some great examples out there. In this article I explain different scenarios of where and how reuse could work.

Don’t reinvent the wheel

There is this ongoing discussion in the PHP community (and I guess in every software-related community) about reinventing wheels. A refreshing angle in this debate came from an article by Phil Sturgeon pointing to the high number of “duplicate” packages available on Packagist. I agree with Phil:

Sometimes these are carbon copies of other packages, but often they are feature-weak versions of established packages.

It doesn’t make sense to do the same thing over and over again. At least I personally don’t try to make this mistake. If I want to write code that “already exists”, at least I don’t publish it on Packagist.

This is a response to Peter Petermann’s article Composer and virtual packages. First, let’s make this totally clear: I don’t want to start an Internet war about this, I’m just pointing out some design issues that may arise from using Composer’s provide option in your package’s composer.json file. This means it’s also nothing personal. To Peter: you wrote a very nice article and shed light on an underused feature of Composer. Thank you for that!

The problem

With a new bundle release you may want to rename services or parameters, make a service private, change some constructor arguments, change the structure of the bundle configuration, etc. Some of these changes may acually be backwards incompatible changes for the users of that bundle. Luckily, the Symfony DependenyInjection component and Config component both provide you with some options to prevent such backwards compatibility (BC) breaks. If you want to know more about backwards compatibility and bundle versioning, please read my previous article on this subject.

A short introduction to semantic versioning

Semantic versioning is an agreement between the user of a package and its maintainer. The maintainer should be able to fix bugs, add new features or completely change the API of the software they provide. At the same time, the user of the package should not be forced to make changes to their own project whenever a package maintainer decides to release a new version.

Symfony bundles: providing services and exposing resources

When you look at the source code of the Symfony framework, it becomes clear that bundles play two distinct and very different roles: in the first place a bundle is a service container extension: it offers ways to add, modify or remove service definitions and parameters, optionally by means of bundle configuration. This role is represented by the following methods of BundleInterface:

namespace Symfony\Component\HttpKernel\Bundle;

interface BundleInterface extends ContainerAwareInterface
{
    /** Boots the Bundle. */
    public function boot();

    /** Shutdowns the Bundle. */
    public function shutdown();

    /** Builds the bundle. */
    public function build(ContainerBuilder $container);

    /** Returns the container extension that should be implicitly loaded. */
    public function getContainerExtension();

    ...
}

The second role of a bundle is that of a resource provider. When a bundle is registered in the application kernel, it automatically starts to expose all kinds of resources to the application. Think of routing files, controllers, entities, templates, translation files, etc.

About interface segregation, dependency inversion and package stability

You are creating a nice reusable package. Inside the package you want to use events to allow others to hook into your own code. You look at several event managers that are available. Since you are somewhat familiar with the Symfony EventDispatcher component already, you decide to add it to your package’s composer.json:

{
    "name": "my/package"
    "require": {
        "symfony/event-dispatcher": "~2.5"
    }
}

Your dependency graph now looks like this:

Seven months, two presentations and three blog posts later, I’ve published the first installment of my new book, Principles of PHP Package Design.

From the introduction of the book:

Naturally the biggest part of this book covers package design principles. But before we come to that, we first take a close look at what constitutes packages: classes and interfaces. The way you design them has great consequences for the characteristics of the package in which they will eventually reside. So before considering package design principles themselves, we first need to take a look at the principles that govern class design. These are the so-called SOLID principles. Each letter of this acronym stands for a different principle, each of which we will briefly (re)visit in the first part of this book.

On several occasions I have tried to explain my opinion about “optional dependencies” (also known as “suggested dependencies” or “dev requirements”) and I’m doing it again:

There’s no such thing as an optional dependency.

I’m talking about PHP packages here and specifically those defined by a composer.json file.

What is a dependency?

First let’s make sure we all agree about what a dependency is. Take a look at the following piece of code:

About the nature of PHP and its misuse in package design

This text will be part of the introduction of my upcoming book Principles of PHP Package Design. If you’d like to be notified when the book is released, you can subscribe on the book’s page. This will entitle you to a 20% discount once the book is available for download.

Before we continue with the “main matter” of this book, I’d like to introduce you to the mindset I had while writing it. This particular mindset amounts to having a constant awareness that PHP has somewhat of a dark side, as well as keeping our hopes high that it is very well possible to expose this “dark side” to a warm sun of programming principles, coding standards and generally good ideas, collected over the years, carefully nurtured and enhanced by great programmers of different languages and cultures.

Recently I tweeted about phpclasses.org. It was not such a friendly statement:

Why does phpclasses.org still exist? Most of the “packages” contain dangerous, stupid or useless code.

Manuel Lemos, the man behind PHP Classes, made me pull back a bit by pointing out that I was generalizing and that they do what they can to encourage people to do a good job. I recognize their effort. And of course, there is also good code on phpclasses.org.

There’s a much more detailed chapter about this subject in my book Principles of PHP Package Design.

These days, good PHP object-oriented libraries are all around and easily available. To me, it is actually thrilling to be part of this flourishing community, while working with Symfony2 and blogging about the Framework, the Components and their neighbors (like Silex). It seems like everything is made for contributing to this nice and friendly environment, with tools like GitHub (online collaboration), Composer (dependency management), Packagist (package archive) and Travis CI (continuous integration).

Several years ago, when the PHP-FIG (PHP Framework Interop Group) created its first PSRs (PHP Standard Recommendations) they started some big changes in the PHP ecosystem. The standard for class auto-loading was created to go hand-in-hand with the then new package manager Composer. PSRs for coding standards were defined, which I’m sure helped a lot of teams to leave coding standard discussions behind. The old tabs versus spaces debate was forever settled and the jokes about it now feel quite outdated.

Several years ago, when the PHP-FIG (PHP Framework Interop Group) created its first PSRs (PHP Standard Recommendations) they started some big changes in the PHP ecosystem. The standard for class auto-loading was created to go hand-in-hand with the then new package manager Composer. PSRs for coding standards were defined, which I’m sure helped a lot of teams to leave coding standard discussions behind. The old tabs versus spaces debate was forever settled and the jokes about it now feel quite outdated.

During a recent run of my Advanced Web Application Architecture training, we discussed the distinction between infrastructure code and non-infrastructure code, which I usually call core code. One of the participants summarized the difference between the two as: “everything in your vendor directory is infrastructure code”. I don’t agree with that, and I will explain why in this article.

Not all code in vendor is infrastructure code

Admittedly, it’s easy for anyone to not agree with a statement like this, because you can simply make up your own definitions of “infrastructure” that turn the statement false. As a matter of fact, I’m currently working on my next book (which has the same title as the training), and I’m working on a memorable definition that covers all the cases. I’ll share with you the current version of that definition, which consists of two rules defining core code. Any code that doesn’t follow both these rules at the same time, should be considered infrastructure code.

@ediar asked me on Twitter if I still think a constructor should not be tested. It depends on the type of object you’re working with, so I think it’ll be useful to elaborate here.

Would you test the constructor of a service that just gets some dependencies injected? No. You’ll test the behavior of the service by calling one of its public methods. The injected dependencies are collaborating services and the service as a whole won’t work if anything went wrong in the constructor.

One of my pet peeves is using the database schema to validate data.

Several ways in which this normally happens:

• Specifying a column as “required”, e.g. email VARCHAR(255) NOT NULL
• Adding an index to force column values to be unique (e.g. CREATE UNIQUE INDEX email_idx ON users(email))
• Adding an index for foreign key integrity, including cascading deletes, etc.

Yes, I want data integrity too. No, I don’t want to rely on the database for that.

The introduction of Domain-Driven Design (DDD) to a larger audience has led to a few really damaging ideas among developers, like this one (maybe it’s more a sentiment than an idea):

Data is bad, behavior is good. The domain model is great, the database awful.

(We’re not even discussing CRUD in this article, which apparently is the worst of the worst.)

By now many of us feel ashamed of using an ORM alongside a “DDD domain model”, putting some mapping configuration in it, doing things inside your entities (or do you call them aggregates?) just to make them easily serializable to the database.

Cyrille Martraire: Domain modeling towards First Principles

This was a talk from the first day, but it required some more processing before writing about it. Cyrille is one of my favorite speakers. He’s fast, funny and brings a lot of interesting topics to the table. So many that it’s sometimes hard to keep following his train of thought, and writing down some notes at the same time.

A central concept from his talk was what he called the waterline between IT and the business. In a traditional scenario, developers get provided with “work” on a case-by-case basis. They don’t learn about the general idea or plan, or even the vision, goal or need that’s behind the “work item”. They just have to “implement” it. It leads to badly designed code. But it also leads to the wrong solutions being delivered. If only developers could have talked with the people who actually have the problem for which they build the solution. Maybe there’s another problem behind it, or maybe the business has provided the developer with a solution, instead of a problem. To higher the waterline means to get more involved with the customers and users, to understand their problems, and work together on a solution. Make sure you get involved.

Eric Evans: Keynote (“Language in Context”)

Starting out with the basics (words have meaning within a context; when we make the boundary of this context explicit we end up with a bounded context), Eric discussed two main topics: the Big Ball of Mud, and bounded contexts in the context (no pun intended) of microservices.

Legacy applications are always framed in a negative way, as if it’s something to get away from. Personally, I’ve come to enjoy them a lot. However, there will always be the urge to work around legacy code. The Bubble Context (PDF) can be a great way of creating a new model that works well next to the already existing models. To keep a safe buffer between the new and the old model, you could build an Anti-Corruption Layer (ACL). A fun thing Eric mentioned is that the buffer works in two directions. The ACL also allows the old model to keep functioning without being disturbed by all the new things that are going on in the Bubble Context.

Last week I wrote about when to add an interface to a class. The article finishes with the claim that classes from the application’s domain don’t usually need an interface. The reason is that domain code isn’t going to be swapped out with something else. This code is the result of careful modelling work that’s done based on the business domain that you work with. And even if you’d work on, say, two financial software projects in a row, you’ll find that the models you produce for each of them will be different in many subtle (if not radical) ways. Paradoxically you’ll find that in practice a domain model can sometimes be reused after all. There are some great examples out there. In this article I explain different scenarios of where and how reuse could work.

In “Negative Architecture”, Michael Feathers speaks about certain aspects of software architecture leading to some kind of negative architecture. Feathers mentions the IO Monad from Haskell (functional programming) as an example, but there are parallel examples in object-oriented programming. For example, by using layers and the dependency inversion principle you can “guarantee” that a certain class, in a certain layer (e.g. domain, application) won’t do any IO - no talking to a database, no HTTP requests to some remote service, etc.

Consider the following rule:

When you create an object, it should be complete, consistent and valid in one go.

It is derived from the more general principle that it should not be possible for an object to exist in an inconsistent state. I think this is a very important rule, one that will gradually lead everyone from the swamps of those dreaded “anemic” domain models. However, the question still remains: what does all of this mean?

The downsides of starting with the domain model

All the architectural focus on having a clean and infrastructure-free domain model is great. It’s awesome to be able to develop your domain model in complete isolation; just a bunch of unit tests helping you design the most beautiful objects. And all the “impure” stuff comes later (like the database, UI interactions, etc.).

However, there’s a big downside to starting with the domain model: it leads to inside-out development. The first negative effect of this is that when you start with designing your aggregates (entities and value objects), you will definitely need to revise them when you end up actually using them from the UI. Some aspects may turn out to be not so well-designed at all, and will make no sense from the user’s perspective. Some functionality may have been designed well, but only theoretically, since it will never actually be used by any real client, except for the unit test you wrote for it.

I’d like to start this article with a quote from Ross Tuck’s article “Persisting Value Objects in Doctrine”. He describes different ways of persisting value objects when using Doctrine ORM. At the end of the page he gives us the following option - the “nuclear” one:

[…] Doctrine is great for the vast majority of applications but if you’ve got edge cases that are making your entity code messy, don’t be afraid to toss Doctrine out. Setup an interface for your repositories and create an alternate implementation where you do the querying or mapping by hand. It might be a PITA but it might also be less frustration in the long run.

This is a question that always pops up during my workshops: when and where to determine the ID of an entity? There are different answers, no best answer. Well, there are two best answers, but they apply to two different situations.

Auto-incrementing IDs, by the database

Traditionally, all you need for an entity to have an ID is to designate one integer column in the database as the primary key, and mark it as “auto-incrementing”. So, once a new entity gets persisted as a record in the database (using your favorite ORM), it will get an ID. That is, the entity has no identity until it has been persisted. Even though this happens everywhere, and almost always; it’s a bit weird, because:

Exceptions - for exceptional situations?

From the Domain-Driven Design movement we’ve learned to go somewhat back to the roots of object-oriented design. Designing domain objects is all about offering meaningful behavior and insights through a carefully designed API. We know now that domain objects with setters for every attribute will allow for the objects to be in an inconsistent state. By removing setters and replacing them with methods which only modify the object’s state in valid ways, we can protect an object from violating domain invariants.

In this series, we’ve discussed several topics already. We talked about persistence and time, the filesystem and randomness. The conclusion for all these areas: whenever you want to “mock” these things, you may look for a solution at the level of programming tools used (use database or filesystem abstraction library, replace built-in PHP functions, etc.). But the better solution is always: add your own abstraction. Start by introducing your own interface (including custom return types), which describes exactly what you need. Then mock this interface freely in your application. But also provide an implementation for it, which uses “the real thing”, and write an integration test for just this class.

I recently came across two interesting methods that were part of a bigger class that I had to redesign:

class OrderLine
{
    /**
     * @var float
     */
    private $quantityOrdered;
    
    // ...

    /**
     * @param float $quantity
     */
    public function processDelivery($quantity)
    {
        $this->quantityDelivered += $quantity;
        
        $this->quantityOpen = $this->quantityOrdered - $quantity;
        if ($this->quantityOpen < 0) {
            $this->quantityOpen = 0;
        }
    }

    /**
     * @param float $quantity
     */
    public function undoDelivery($quantity)
    {
        $this->quantityDelivered -= $quantity;
        if ($this->quantityDelivered < 0) {
            $this->quantityDelivered = 0;
        }
        
        $this->quantityOpen += $quantity;
        if ($this->quantityOpen > $this->quantityOrdered) {
            $this->quantityOpen = $this->quantityOrdered;
        }
    }
}

Of course, I’ve already cleaned up the code here to allow you to better understand it.

In previous articles I’ve already spent a few words on some of the talks I watched at DDD Europe that made quite an impact on my thoughts about software development and design. The design part is often the most philosophical one. Design talks are often less practical. Speakers use metaphors to bring their message across and take note of other disciplines than software engineering to discover useful design principles and heuristics.

Rebecca Wirfs-Brock: Design Matters

The opening keynote of the second day was particularly philosophical. To be honest, I didn’t know Rebecca. She is co-author of several important books on object design: “Designing Object-Oriented Software” and “Object Design: Roles, Responsibilities, and Collaborations”. It should be interesting to take a look at these some time.

In a previous post I discussed some of the talks I attended at the DDD Europe conference in Amsterdam. This conference has offered a lot more amazing content and I’d like to continue to tell you about it.

Lately I’ve been thinking about why Domain-Driven Design (DDD) has such a great attractive force on me. In particular when at the same time I’m strongly attracted by technology like Docker and its surrounding ecosystem of tools that are finally helping us to fulfill the ideal of continuous delivery, with relative ease. Configuring servers, deploying them, connecting them, etc. is what we previously would have called “operations” work. Now that developers themselves get to do more and more operations work, a movement was called to life: devops, where two previously separated responsibilities are merged into one: writing software and running it in production is part of the same, single task: shipping working software.

I’m currently attending the DDD Europe conference in Amsterdam and thought it might be interesting for people at home to read up on some of the topics that this interesting (and well-organized) conference covers.

Mel Conway: Consider the development feedback loop

On Twitter there were outbursts of “The legend on stage” and “This is the Conway of ‘Conway’s Law’” when Mel Conway entered the stage. I knew about the law, which amounts to the observation that software systems tend to be shaped by the way the teams working on it communicate.

Visiting Qandidate in Rotterdam

Last Thursday I had the honor of meeting (part of the) Broadway team: three very smart developers working for Qandidate in central Rotterdam: Gediminas Šedbaras, Willem-Jan Zijderveld and Fritsjan. A short walk from the train station brought me to their office, where we talked about CQRS, event sourcing and some of the practical aspects of using the Broadway framework.

As you may have read I’ve been experimenting with Broadway extensively during these last weeks. Knowing a lot about the theory behind it, it was really nice to see how everything worked so smoothly. I had some questions however, which have now been answered (thank you guys for taking the time to do this!).

Event sourcing with Broadway

At the Dutch PHP Conference I attended a workshop by Beau Simensen and Willem-Jan Zijderveld. They showed us some examples of how to work with Broadway, a framework for event sourcing, with full Symfony integration, created by the smart people at Qandidate.

During my two weeks of funemployment, before starting my new job at Ibuildings, I decided to recreate one of my previous projects using Broadway. As it turns out, it’s a great framework that’s quite easy to use and is very powerful at the same time. Even though it’s not a stable package (as in, it’s still in the 0.x version range), I think it’s safe to depend on it.

An ever recurring pattern in my life is this one:

1. I stumble upon some interesting piece of code, an intriguing book chapter, a fascinating concept, etc.
2. Slowly, over the next couple of weeks, my brain realises that, yes, this is some very interesting stuff.
3. Then I want to become all productive about it - writing things on my blog, speaking about it in public, maybe even writing a book about it.

This time it was domain-driven design (DDD), command-query responsibility segregation (CQRS) and in particular its architectural and technical aspects. While playing with existing libraries I soon recognized the huge benefits of applying hexagonal architecture and some of the tactical DDD patterns to a (Symfony) codebase.

@ediar asked me on Twitter if I still think a constructor should not be tested. It depends on the type of object you’re working with, so I think it’ll be useful to elaborate here.

Would you test the constructor of a service that just gets some dependencies injected? No. You’ll test the behavior of the service by calling one of its public methods. The injected dependencies are collaborating services and the service as a whole won’t work if anything went wrong in the constructor.

This article is an excerpt from my book Advanced Web Application Architecture. It contains a couple of sections from the conclusion of Part I: Decoupling from infrastructure.

This chapter covers:

• A deeper discussion on the distinction between core and infrastructure code
• A summary of the strategy for pushing infrastructure to the sides
• A recommendation for using a domain- and test-first approach to software development
• A closer look at the concept of “pure” object-oriented programming

Core code and infrastructure code

In Chapter 1 we’ve looked at definitions for the terms core code and infrastructure code. What I personally find useful about these definitions is that you can look at a piece of code and find out if the definitions apply to it. You can then decide if it’s either core or infrastructure code. But there are other ways of applying these terms to software. One way is to consider the bigger picture of the application and its interactions with actors. You’ll find the term actor in books about user stories and use cases by authors like Ivar Jacobson and Alistair Cockburn, who make a distinction between:

I write about design rules a lot, but I sometimes forget to:

1. Mention that these rules can’t always be applied,
2. Describe when that would be the case, and
3. Add examples of situations where the rule really doesn’t matter.

The rules should work in most cases, but sometimes need to be “violated”. Which is too strong a word anyway. When someone points out to me that I violated a rule, I’m like: Wow! I violated the rule? I’m so sorry! Let’s fix this immediately. Whereas in practice it should be more like: Yeah, I know that rule, but it makes more sense to follow that other rule here, because […]. In other words, pointing out that a certain rule has been violated should not be a sufficient reason to adhere to that rule. My favorite example is “But that violates SRP!” (Single Responsibility Principle). Whoops, I wouldn’t want to do that! Or would I?

I recently heard this interesting question: if your project uses a static analysis tool like PHPStan or Psalm (as it should), should the tests by analysed too?

The first thing to consider: what are potential reasons for not analysing your test code?

Why not?

1. Tests are messy in terms of types

Tests may use mocks, which can be confusing for the static analyser:

$object = $this->createMock(MockedInterface::class);

The actual type of $object is an intersection type, i.e. $object is both a MockObject and a MockedInterface, but the analyser only recognizes MockObject. You may not like all those warnings about “unknown method” calls on MockObject $object so you exclude test code from the analysis.

A lot can happen in 9 years. Back then I was still advocating that you should unit-test your controllers and that setter injection is very helpful when replacing controller dependencies with test doubles. I’ve changed my mind: constructor injection is the right way for any service object, including controllers. And controllers shouldn’t be unit tested, because:

• Those unit tests tend to be a one-to-one copy of the controller code itself. There is no healthy distance between the test and the implementation.
• Controllers need some form of integrated testing, because by zooming in on the class-level, you don’t know if the controller will behave well when the application is actually used. Is the routing configuration correct? Can the framework resolve all of the controller’s arguments? Will dependencies be injected properly? And so on.

The alternative I mentioned in 2012 is to write functional tests for your controller. But this is not preferable in the end. These tests are slow and fragile, because you end up invoking much more code than just the domain logic.

Thanks! Let me explain myself

First of all: thanks everybody for reading the previous parts of this series. Many people posted some interesting comments. I realized quickly that I had to explain myself a bit: why am I writing all of this? Why would you ever want to decouple controllers from the (Symfony2) framework? You probably don’t need to bother anyway, because

The chances of you needing to move controllers to other frameworks is next to none. — Rafael Dohms

In the previous part of this series we decreased coupling of a Symfony controller to the Symfony2 framework by removing its dependency on the standard Controller class from the FrameworkBundle.

Now we take a look at annotations. They were initially introduced for rapid development (no need to create/modify some configuration file, just solve the issues inline!):

namespace Matthias\ClientBundle\Controller;

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Method;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\ParamConverter;

/**
 * @Route("/client")
 */
class ClientController
{
    /**
     * @Route('/{id}')
     * @Method("GET")
     * @ParamConverter(name="client")
     * @Template
     */
    public function detailsAction(Client $client)
    {
        return array(
            'client' => $client
        );
    }
}

When you use these annotations, the details action will be executed when the URL matches /client/{id}. A param converter will be used to fetch a Client entity from the database based on the id parameter which is extracted from the URL by the router. The return value of the action is an array. These are used as template variables for rendering the Resources/views/Client/Details.html.twig template.

Part I: Don’t use the standard controller

The general belief is that controllers are the most tightly coupled classes in every application. Most of the time based on the request data, they fetch and/or store persistent data from/in some place, then turn the data into HTML, which serves as the response to the client who initially made the request.

So controllers are “all over the place”, they glue parts of the application together which normally lie very far from each other. This would make them highly coupled: they depend on many different things, like the Doctrine entity manager, the Twig templating engine, the base controller from the FrameworkBundle, etc.

There’s a much more detailed chapter about this subject in my book A Year With Symfony.

Symfony2 provides multiple ways of blocking, providing or modifying the response. You can:

1. Intercept each request by listening to the kernel.request event and set the response directly on the event (which will effectively skip execution of a controller)

2. Modify the controller or its arguments by listening to the kernel.controller event, then calling setController on the event object and modifying the attributes of the Request object.

In my previous post I wrote about uploading files to GridFS. Therefor I created a MongoDB Document with a $file property annotated with @MongoDB\File. Because I am using ORM entities more often then ODM documents, I was looking for a seamless way to access a Document from an Entity.

Because it isn’t possible to define a direct relationship between an Entity and a Document I thought it would be a solid solution to create a custom field type. By defining a custom field type I can control the way the reference to the Document will be stored and at the same time I will be able to restore the reference when retrieving the field. The steps needed to create a custom field type for ORM entities are very similar to the post of Matthias on how to create custom field types for ODM documents.

Almost at the same time, I silently celebrate the first birthday of my blog. My first article appeared a little over a year ago. It is great to see how Symfony2 has become more and more popular during these twelve months. Your comments and visits encourage me to keep posting articles. So, thank you all! And thanks, Dennis, for contributing.

GridFS is a specification for storing large files in MongoDB. In this post I will explain how you can easily upload a file to GridFS and then retrieve it from the database to serve it to the browser.

Ever since I am using the Symfony Framework (be it version 1 or 2), I tend to describe every other project I’ve done (including those that were built on top of some third party “framework” like Joomla or WordPress) as a “legacy project”. Though this has sometimes felt like treason, I still keep doing it: the quality of applications written using Symfony is usually so much higher in terms of maintainability, security and code cleanliness, that even a project done last year using “only PHP” looks like a mess and seems to be no good software at all. So I feel the strong urge to rebuild everything I have in portfolio (as do many other developers), but “this time, I will do it the right way”.

Apparently not everyone agrees on how to unit test their Symfony2 controllers. Some treat controller code as the application’s “glue”: a controller does the real job of transforming a request to a response. Thus it should be tested by making a request and check the received response for the right contents. Others treat controller code just like any other code - which means that every path the interpreter may take, should be tested.

When you don’t have access to the command-line of your webserver, it may be nice to still run all your unit tests; so you need a way to execute the phpunit command from within a controller. This way, you can call your test suite by browsing to a URL of your site. To do things right, we start with a “test” controller /web/app_test.php containing these lines of code:

if (!in_array(@$_SERVER['REMOTE_ADDR'], array(
    '127.0.0.1',
    '::1',
))) {
    header('HTTP/1.0 403 Forbidden');
    exit('You are not allowed to access this file. Check '.basename(__FILE__).' for more information.');
}

require_once __DIR__.'/../app/bootstrap.php.cache';
require_once __DIR__.'/../app/AppKernel.php';

use Symfony\Component\HttpFoundation\Request;

$kernel = new AppKernel('test', true);
$kernel->loadClassCache();
$kernel->handle(Request::createFromGlobals())->send();

In fact, just copy everything from /web/app_dev.php but replace the string “dev” by “test”.

Where should it go?

If you’re one of those people who make a separation between an application and a domain layer in their code base (like I do), then a question you’ll often have is: does this service go in the application or in the domain layer? It sometimes makes you wonder if the distinction between these layers is superficial after all. I’m not going to write again about what the layers mean, but here is how I decide if a service goes into Application or Domain:

Since I’ve been writing a lot about decoupled application development it made sense that one of my readers asked the following question: “Why should we use a framework?” The quick answer is: because you need it. A summary of the reasons:

• It would be too much work to replace all the work that the framework does for you with code written by yourself. Software development is too costly for this.
• Framework maintainers have fixed many issues before you even encountered them. They have done everything to make the code secure, and when a new security issue pops up, they fix it so you can just pull the latest version of the framework.
• By not using a framework you will be decoupled from Symfony, Laravel, etc. but you will be coupled to Your Own Framework, which is a bigger problem since you’re the maintainer and it’s likely that you won’t actually maintain it (in my experience, this is what often happens to projects that use their own home-grown framework).

So, yes, you/we need a framework. At the same time you may want to write framework-decoupled code whenever possible.

100% done

I’m happy to announce that my latest book “Advanced Web Application Architecture” is now complete. With ~390 pages it has become a well-rounded book full of useful design patterns and architectural principles built around the notion of object-pure code, that will help you create decoupled applications that are testable by definition, and support a domain-first approach to software development.

Use this link to get 10% off the price: https://leanpub.com/web-application-architecture/c/RELEASE_DAY

In the Epilogue of the Object Design Style Guide, I started happily outlining some of the architectural patterns I’ve been using for several years now. I wanted to give some kind of an overview of how the overall design of your application would improve if you apply the object-design rules in that book. I soon realized that an Epilogue was not enough to cover all the details, or to communicate the ideas in such a way that they would be applicable in everyday projects. And so a new book project began…

During a recent run of my Advanced Web Application Architecture training, we discussed the distinction between infrastructure code and non-infrastructure code, which I usually call core code. One of the participants summarized the difference between the two as: “everything in your vendor directory is infrastructure code”. I don’t agree with that, and I will explain why in this article.

Not all code in vendor is infrastructure code

Admittedly, it’s easy for anyone to not agree with a statement like this, because you can simply make up your own definitions of “infrastructure” that turn the statement false. As a matter of fact, I’m currently working on my next book (which has the same title as the training), and I’m working on a memorable definition that covers all the cases. I’ll share with you the current version of that definition, which consists of two rules defining core code. Any code that doesn’t follow both these rules at the same time, should be considered infrastructure code.

In this series, we’ve discussed several topics already. We talked about persistence and time, the filesystem and randomness. The conclusion for all these areas: whenever you want to “mock” these things, you may look for a solution at the level of programming tools used (use database or filesystem abstraction library, replace built-in PHP functions, etc.). But the better solution is always: add your own abstraction. Start by introducing your own interface (including custom return types), which describes exactly what you need. Then mock this interface freely in your application. But also provide an implementation for it, which uses “the real thing”, and write an integration test for just this class.

In a previous article, we discussed “persistence” and “time” as boundary concepts that need mocking by means of dependency inversion: define your own interface, then provide an implementation for it. There were three other topics left to cover: the filesystem, the network and randomness.

Mocking the filesystem

We already covered “persistence”, but only in the sense that we sometimes need a way to make in-memory objects persistent. After a restart of the application we should be able to bring back those objects and continue to use them as if nothing happened.

I read this tweet:

"The object-oriented version of spaghetti code is, of course, 'lasagna code'. Too many layers." - Roberto Waltman

— Programming Wisdom (@CodeWisdom) February 24, 2018

Jokes taken as advice

It’s not the first time I’d heard of this quote. Somehow it annoys me, not just this one joke, but many jokes like this one. I know I should be laughing, but I’m always worried about jokes like this going to be interpreted as advice, in its most extreme form. E.g. the advice distilled from this tweet could be: “Layers? Don’t go there. Before you know it, you have lasagna code…”

More and more I’ve come to realize that I’ve been mocking less and less.

The thing is, creating test doubles is a very dangerous activity. For example, what I often see is something like this:

$entityManager = $this->createMock(EntityManager::class);

$entityManager->expects($this->once())
    ->method('persist')
    ->with($object);
$entityManager->expects($this->once())
    ->method('flush')
    ->with($object);

Or, what appears to be better, since we’d be mocking an interface instead of a concrete class:

$entityManager = $this->createMock(ObjectManagerInterface::class);
// ...

To be very honest, there isn’t a big different between these two examples. If this code is in, for example, a unit test for a repository class, we’re not testing many of the aspects of the code that should have been tested instead.

In the previous article we discussed a sensible layer system, consisting of three layers:

• Domain
• Application
• Infrastructure

Infrastructure

The infrastructure layer, containing everything that connects the application’s use cases to “the world outside” (like users, hardware, other applications), can become quite large. As I already remarked, a lot of our software consists of infrastructure code, since that’s the realm of things complicated and prone to break. Infrastructure code connects our precious clean code to:

The first key concept of what I think is a very simple, at the very least “clean” architecture, is the concept of a layer. A layer itself is actually nothing, if you think about it. It’s simply determined by how it’s used. Let’s stay a bit philosophical, before we dive into some concrete architectural advice.

Qualities of layers

A layer in software serves the following (more or less abstract) purposes:

Looking back at my old blog posts, I think it’s good to write down a more balanced view on application architecture than the one that speaks from some of the older posts from 2013 and 2014. Before I do, I allow myself a quick self-centered trip down memory lane.

Why Symfony? Seven facts

The archive tells an interesting story about how my thoughts about software development changed over time. It all started with my first post in October 2011, How to make your service use tags. Back in those days, version 2 of the Symfony framework was still quite young. I had been working with symfony (version 1, small caps) for a couple of years and all of a sudden I was using Symfony2, and I absolutely loved this new shiny framework. I learned as much as I could about it (just as I had done with version 1), and eventually I became a Certified Symfony Developer, at the first round of exams held in Paris, during a Symfony Live conference. During those years I blogged and spoke a lot about Symfony, contributed to the documentation, and produced several open source bundles. I also wrote a book about it: A Year With Symfony.

Where should it go?

If you’re one of those people who make a separation between an application and a domain layer in their code base (like I do), then a question you’ll often have is: does this service go in the application or in the domain layer? It sometimes makes you wonder if the distinction between these layers is superficial after all. I’m not going to write again about what the layers mean, but here is how I decide if a service goes into Application or Domain:

I read this tweet:

"The object-oriented version of spaghetti code is, of course, 'lasagna code'. Too many layers." - Roberto Waltman

— Programming Wisdom (@CodeWisdom) February 24, 2018

Jokes taken as advice

It’s not the first time I’d heard of this quote. Somehow it annoys me, not just this one joke, but many jokes like this one. I know I should be laughing, but I’m always worried about jokes like this going to be interpreted as advice, in its most extreme form. E.g. the advice distilled from this tweet could be: “Layers? Don’t go there. Before you know it, you have lasagna code…”

This is an excerpt from my book PHP for the Web. It’s a book for people who want to learn to build web applications with PHP. It doesn’t focus on PHP programming, but shows how PHP can be used to serve dynamic web pages. HTTP requests and responses, forms, cookies, and sessions. Use it all to build a CRUD interface and an authentication system for your very first web application.

Chapter 11: Error handling

This is an excerpt from my book PHP for the Web. It’s a book for people who want to learn to build web applications with PHP. It doesn’t focus on PHP programming, but shows how PHP can be used to serve dynamic web pages. HTTP requests and responses, forms, cookies, and sessions. Use it all to build a CRUD interface and an authentication system for your very first web application.

Chapter 11: Error handling

Exceptions - for exceptional situations?

From the Domain-Driven Design movement we’ve learned to go somewhat back to the roots of object-oriented design. Designing domain objects is all about offering meaningful behavior and insights through a carefully designed API. We know now that domain objects with setters for every attribute will allow for the objects to be in an inconsistent state. By removing setters and replacing them with methods which only modify the object’s state in valid ways, we can protect an object from violating domain invariants.

You’ve picked a good refactoring goal. You are prepared to stop the project at anytime. Now how to determine the steps that lead to the goal?

Bottom-up development

There is an interesting similarity between refactoring projects, and regular projects, where the goal is to add some new feature to the application. When working on a feature, I’m always happy to jump right in and think about what value objects, entities, controllers, etc. I need to build. Once I’ve written all that code and I’m ready to connect the dots, I often realize that I have created building blocks that I don’t even need, or that don’t offer a convenient API. This is the downside of what’s commonly called “bottom-up development”. Starting to build the low-level stuff, you can’t be certain if you’re contributing to the higher-level goal you have in mind.

Refactoring is often mentioned in the context of working with legacy code. Maybe you like to define legacy code as code without tests, or code you don’t understand, or even as code you didn’t write. Very often, legacy code is code you just don’t like, whether you wrote it, or someone else did. Since the code was written the team has introduced new and better ways of doing things. Unfortunately, half of the code base still uses the old and deprecated way…

Refactoring projects

A common case of refactoring-gone-wrong is when refactoring becomes a large project in a branch that can never be merged because the refactoring project is never completed. The refactoring project is considered a separate project, and soon starts to feel like “The Big Rewrite That Always Fails” from programming literature.

The work happens in a branch because people actually fear the change. They want to see it before they believe it, and review every single part of it before it can be merged. This process may take months. Meanwhile, other developers keep making changes to the main branch, so merging the refactoring branch is going to be a very tedious, if not dangerous thing to do. A task that, on its own, can cause the failure of the refactoring project itself.

Statically fetching dependencies

I’ve worked with several code bases that were littered with calls to Zend_Registry::get(), sfContext::getInstance(), etc. to fetch a dependency when needed. I’m a little afraid to mention façades here, but they also belong in this list. The point of this article is not to bash a certain framework (they are all lovely), but to show how to get rid of these “centralized dependency managers” when you need to. The characteristics of these things are:

I find it very curious that legacy (PHP) code often has the following characteristics:

1. Classes with the name of a central domain concept have grown too large.
2. Methods in these classes have become very generic.

Classes grow too large

I think the following happened:

The original developers tried to capture the domain logic in these classes. They implemented it based on what they knew at the time. Other developers, who worked on the code later, had to implement new features, or modify domain logic, because, well, things change. Also, because we need more things.

This article continues where Unary call sites and intention-revealing interfaces ended.

While reading David West’s excellent book “Object Thinking”, I stumbled across an interesting quote from David Parnas on the programming method that most of us use by default:

The easiest way to describe the programming method used in most projects today was given to me by a teacher who was explaining how he teaches programming. “Think like a computer,” he said. He instructed his students to begin by thinking about what the computer had to do first and to write that down. They would then think about what the computer had to do next and continue in that way until they had described the last thing the computer would do… […]

Call sites

One of the features I love most about my IDE is the button “Find Usages”. It is invaluable when improving a legacy code base. When used on a class it will show you where this class is used (as a parameter type, in an import statement, etc.). When used on a method, it will show you where this method gets called. Users of a method are often called “clients”, but when we use “Find Usages”, we might as well use the more generic term “call sites”.

Setting the stage: Code complexity

Code complexity often gets measured by calculating the Cyclomatic Complexity per unit of code. The number can be calculated by taking all the branches of the code into consideration.

Code complexity is an indicator for several things:

• How hard it is to understand a piece of code; a high number indicates many branches in the code. When reading the code, a programmer has to keep track of all those branches, in order to understand all the different ways in which the code can work.
• How hard it is to test that piece of code; a high number indicates many branches in the code, and in order to fully test the piece of code, all those branches need to be covered separately.

In both cases, high code complexity is a really bad thing. So, in general, we always strive for low code complexity. Unfortunately, many projects that you’ll inherit (“legacy projects”), will contain code that has high code complexity, and no tests. A common hypothesis is that a high code complexity arises from a lack of tests. At the same time, it’s really hard to write tests for code with high complexity, so this is a situation that is really hard to get out.

CQRS - not a complicated thing

CQRS has some reputation issues. Mainly, people will feel that it’s too complicated to apply in their current projects. It will often be considered over-engineering. I think CQRS is simply misunderstood, which is the reason many people will not choose it as a design technique. One of the common misconceptions is that CQRS always goes together with event sourcing, which is indeed more costly and risky to implement.

Leaving Qandidate, off to Coolblue

After I had a very interesting conversation with the developers behind the Broadway framework for CQRS and event sourcing the day wasn’t over for me yet. I walked about one kilometer to the north to meet Paul de Raaij, who is a senior developer at Coolblue, a company which sells and delivers all kinds of - mostly - electrical consumer devices. Their headquarters are very close to the new and shiny Rotterdam Central station. The company itself is doing quite well. With 1000+ employees they keep needing more office space.

Legacy code

I’m happy that I’ve discovered the work of P.M. Jones recently. His and mine interests seem to align at several interesting points. Though I don’t personally enjoy “putting .308 holes in targets at 400 yards” (as quoted by Phil Sturgeon), I do care a great deal about package coupling (and cohesion for that matter) and I’m also lightly inflammable when it comes to the use of service locators. It also appears that Paul, just like myself and many others in this business, has felt the pain of working on a legacy PHP application, trying to add features to it, or change existing behavior. This is a particularly hard thing to do and because so many incompetent developers have been creating PHP applications since the dawn of PHP, chances are that a big part of the job of competent PHP developers nowadays consists of maintaining these dumpsites of include statements and superglobals.

In my previous post, I wrote about wrapping a legacy application in Silex, using output buffering and Twig. Finally, to allow for better decoupling as well as lazy loading of services, we passed the actual Silex\Application instance as the first argument of legacy controllers.

The first and quite easy way we can enhance our legacy application, is to make use of the request service (which contains all the details about the current request, wrapped inside the Symfony HttpFoundation’s Request class). So, instead of reading directly from $_GET and $_POST, we can change the edit_category() controller into the following:

Since I’ve been writing a lot about decoupled application development it made sense that one of my readers asked the following question: “Why should we use a framework?” The quick answer is: because you need it. A summary of the reasons:

• It would be too much work to replace all the work that the framework does for you with code written by yourself. Software development is too costly for this.
• Framework maintainers have fixed many issues before you even encountered them. They have done everything to make the code secure, and when a new security issue pops up, they fix it so you can just pull the latest version of the framework.
• By not using a framework you will be decoupled from Symfony, Laravel, etc. but you will be coupled to Your Own Framework, which is a bigger problem since you’re the maintainer and it’s likely that you won’t actually maintain it (in my experience, this is what often happens to projects that use their own home-grown framework).

So, yes, you/we need a framework. At the same time you may want to write framework-decoupled code whenever possible.

This is just a quick post sharing something I was able to figure out after doing some research.

The situation: our application throws exceptions by means of “talking back to the user”. As developer we don’t want to be notified about all these exceptions. They aren’t as important as any other exception that should be considered “critical”. Still, we do want to find these exceptions in the logs, because they can sometimes provide valuable feedback about the usability of the system.

There’s a much more detailed chapter about this subject in my book A Year With Symfony.

When working with Symfony2, you already have many of the finest tools for securing your web application. There are cases however that require you to add that extra bit. In this post I will point you to the right extension points within a Symfony2 project (or any other project which uses the Security Component for that matter).

This is just a quick post sharing something I was able to figure out after doing some research.

The situation: our application throws exceptions by means of “talking back to the user”. As developer we don’t want to be notified about all these exceptions. They aren’t as important as any other exception that should be considered “critical”. Still, we do want to find these exceptions in the logs, because they can sometimes provide valuable feedback about the usability of the system.

If you want to write applications that are maintainable in the long run, you have to decouple from your framework, ORM, HTTP client, etc. because your application will outlive all of them.

Three simple rules

To accomplish framework decoupling you only have to follow these simple rules:

1. All services should get all their dependencies and configuration values injected as constructor arguments. When a dependency uses IO, you have to introduce an abstraction for it.
2. Other types of objects shouldn’t have service responsibilities.
3. Contextual information should always be passed as method arguments.

Explanations

Rule 1

Following rule 1 ensures that you’ll never fetch a service ad hoc, e.g. by using Container::get(UserRepository::class). This is needed for framework decoupling because the global static facility that returns the service for you is by definition framework-specific. The same is true for fetching configuration values (e.g. Config::get('email.default_sender')).

One of my pet peeves is using the database schema to validate data.

Several ways in which this normally happens:

• Specifying a column as “required”, e.g. email VARCHAR(255) NOT NULL
• Adding an index to force column values to be unique (e.g. CREATE UNIQUE INDEX email_idx ON users(email))
• Adding an index for foreign key integrity, including cascading deletes, etc.

Yes, I want data integrity too. No, I don’t want to rely on the database for that.

The introduction of Domain-Driven Design (DDD) to a larger audience has led to a few really damaging ideas among developers, like this one (maybe it’s more a sentiment than an idea):

Data is bad, behavior is good. The domain model is great, the database awful.

(We’re not even discussing CRUD in this article, which apparently is the worst of the worst.)

By now many of us feel ashamed of using an ORM alongside a “DDD domain model”, putting some mapping configuration in it, doing things inside your entities (or do you call them aggregates?) just to make them easily serializable to the database.

In part 1 of this short series (it’s going to end with this article) we covered how you can test-drive the queries in a repository class. Returning query results is only part of the job of a repository though. The other part is to store objects (entities), retrieve them using something like a save() and a getById() method, and possibly delete them. Some people will implement these two jobs in one repository class, some like to use two or even many repositories for this. When you have a separate write and read model (CQRS), the read model repositories will have the querying functionality (e.g. find me all the active products), the write model repositories will have the store/retrieve/delete functionality.

There’s something I’ve only been doing since a year or so, and I’ve come to like it a lot. My previous testing experiences were mostly at the level of unit tests or functional/system tests. What was left out of the equation was integration tests. The perfect example of which is a test that proves that your custom repository class works well with the type of database that you use for the project, and possibly the ORM or database abstraction library you use to talk with that database. A test for a repository can’t be a unit test; that wouldn’t make sense. You’d leave a lot of assumptions untested. So, no mocking is allowed.

System and integration tests need database fixtures. These fixtures should be representative and diverse enough to “fake” normal usage of the application, so that the tests using them will catch any issues that might occur once you deploy the application to the production environment. There are many different options for dealing with fixtures; let’s explore some of them.

Generate fixtures the natural way

The first option, which I assume not many people are choosing, is to start up the application at the beginning of a test, then navigate to specific pages, submit forms, click buttons, etc. until finally the database has been populated with the right data. At that point, the application is in a useful state and you can continue with the act/when and assert/then phases. (See the recent article “Pickled State” by Robert Martin on the topic of tests as specifications of a finite state machine).

Something that always bothers me: persistence (the user interface too, but that’s a different topic ;)). Having objects in memory is nice, but when the application shuts down (and for PHP this is after every request-response cycle), you have to persist them somehow. By the way, I think we’ve all forever been annoyed by persistence, since there’s an awful lot of software solutions related to object persistence: different types of databases, different types of ORMs, etc.

CQRS - not a complicated thing

CQRS has some reputation issues. Mainly, people will feel that it’s too complicated to apply in their current projects. It will often be considered over-engineering. I think CQRS is simply misunderstood, which is the reason many people will not choose it as a design technique. One of the common misconceptions is that CQRS always goes together with event sourcing, which is indeed more costly and risky to implement.

I wanted to share with you a free chapter from my latest book, “Advanced Web Application Architecture”. I’ve picked Chapter 11, which gives a compact overview of all the design patterns that are useful for structuring your web application in a way that will (almost) automatically make it independent of surrounding infrastructure, including the web framework you use.

Chapter 11 is the first chapter of Part II of the book. In Part I we’ve been discovering these design patterns by refactoring different areas of a simple web application. Part II provides some higher-level concepts that can help you structure your application. Besides design patterns, it covers architectural layering, and hexagonal architecture (ports & adapters). It also includes a chapter on testing decoupled applications.

100% done

I’m happy to announce that my latest book “Advanced Web Application Architecture” is now complete. With ~390 pages it has become a well-rounded book full of useful design patterns and architectural principles built around the notion of object-pure code, that will help you create decoupled applications that are testable by definition, and support a domain-first approach to software development.

Use this link to get 10% off the price: https://leanpub.com/web-application-architecture/c/RELEASE_DAY

While Advanced Web Application Architecture is still a work in progress, I decided to release another project in the meantime: a book for beginning PHP developers called PHP for the Web.

Of course, there are PHP books for beginners, but most of them aren’t very concise. They cover many related topics like relational databases, CLI applications, and sending emails. I wanted to write a book that only shows what makes PHP so special when it comes to web development.

In the Epilogue of the Object Design Style Guide, I started happily outlining some of the architectural patterns I’ve been using for several years now. I wanted to give some kind of an overview of how the overall design of your application would improve if you apply the object-design rules in that book. I soon realized that an Epilogue was not enough to cover all the details, or to communicate the ideas in such a way that they would be applicable in everyday projects. And so a new book project began…

When Symfony2 was created I first learned about the functional test, which is an interesting type of test where everything about your application is as real as possible. Just like with an integration or end-to-end test. One big difference: the test runner exercises the application’s front controller programmatically, instead of through a web server. This means that the input for the test is a Request object, not an actual HTTP message.

Unbalanced test suites

Having spoken to many developers and development teams so far, I’ve recognized several patterns when it comes to software testing. For example:

1. When the developers use a framework that encourages or sometimes even forces them to marry their code to the framework code, they only write functional tests - with a real database, making (pseudo) HTTP requests, etc. using the test tools bundled with the framework (if you’re lucky, anyway). For these teams it’s often too hard to write proper unit tests. It takes too much time, or is too difficult to set up.
2. When the developers use a framework that enables or encourages them to write code that is decoupled from the framework, they have all these nice, often pretty abstract, units of code. Those units are easy to test. What often happens is that these teams end up writing only unit tests, and don’t supply any tests or “executable specifications” proving the correctness of the behavior of the application at large.
3. Almost nobody writes proper acceptance tests. That is, most tests that use a BDD framework, are still solely focusing on the technical aspects (verifying URLs, HTML elements, rows in a database, etc.), while to be truly beneficial they should be concerned about application behavior, defined in a ubiquitous language that is used by both the technical staff and the project’s stakeholders.

Please note that these are just some very rough conclusions, there’s nothing scientific about them. It would be interesting to do some actual research though. And probably someone has already done this.

Apparently not everyone agrees on how to unit test their Symfony2 controllers. Some treat controller code as the application’s “glue”: a controller does the real job of transforming a request to a response. Thus it should be tested by making a request and check the received response for the right contents. Others treat controller code just like any other code - which means that every path the interpreter may take, should be tested.

It’s very easy not to write a book.

Mostly since it’s such an awful lot of work. You’ll first need to figure out what you’re going to write about, and find an angle that makes it interesting enough for potential readers to buy the book and spend the time to finish it. The writing itself is hard work as well, and may take several years. Along the way you’ll be telling yourself things that will keep you from finishing the task: I’m not doing anything original here. Nobody will be interested. Somebody else wrote a better book about this than I could ever do. If I don’t cover topic B as well, it’s not going to be a useful book at all.

During the holiday I read a book mentioned to me by Pim Elshoff: “Skin in the game”, by Nassim Nicholas Taleb. Discussing this concept of “skin in the game” with Pim had made me curious about the book. It’s not such a fat book as one of Taleb’s other books, which is possibly more famous, called “Antifragile”. “Skin in the game” is a much lighter book, and also quite polemic, making it an often uncomfortable, but fun reading experience. I can easily see how people could get mad about this book or aggressive towards its author (not that I’m encouraging or approving of that aggression!). While reading it, it reminded me of Nietzsche, and his despise of the “common man”, who Taleb calls “the intellectual” - someone who has no “skin in the game”, let alone “soul in the game”. Taleb’s ideas are interesting, just like Nietzsche’s are, but they could easily be abused, and probably so by misinterpreting them.

This article is part 2 of my review of the book “Fifty quick ideas to improve your tests”. I’ll continue to share some of my personal highlights with you.

Replace multiple steps with a higher-level step

If a test executes multiple tasks in sequence that form a higher-level action, often the language and the concepts used in the test explain the mechanics of test execution rather than the purpose of the test, and in this case the entire block can often be replaced with a single higher-level concept.

Review

After reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose, I picked up another book, which I bought a long time ago: “Fifty Quick Ideas to Improve Your Tests” by Gojko Adzic, David Evans, Tom Roden and Nikola Korac. Like with so many books, I find there’s often a “right” time for them. When I tried to read this book for the first time, I was totally not interested and soon stopped trying to read it. But ever since Julien Janvier asked me if I knew any good resources on how to write good acceptance test scenarios, I kept looking around for more valuable pointers, and so I revisited this book too. After all, one of the author’s of this book - Gojko Adzic - also wrote “Bridging the communication gap - Specification by example and agile acceptance testing”, which made a lasting impression on me. If I remember correctly, the latter doesn’t have too much practical advice on writing goods tests (or scenarios), and it was my hope that “Fifty quick ideas” would.

I’ve just finished reading “Discovery - Explore behaviour using examples” by Gáspár Nagy and Seb Rose. It’s the first in a series of books about BDD (Behavior-Driven Development). The next parts are yet to be written/published. Part of the reason to pick up this book was that I’d seen it on Twitter (that alone would not be a sufficient reason of course). The biggest reason was that after delivering a testing and aggregate design workshop, I noticed that my acceptance test skills aren’t what they should be. After several years of not working as a developer on a project for a client, I realized again that (a quote from the book):

Recently I read Ouarzy’s review of Jason Fried and David Heinemeier Hansson’s “Remote - Office Not Required”. I’d read their previous books, “Getting Real” and “Rework”. They’re all a joy to read. Short chapters, nice little cartoons. Just a lot of fun, and inspiring too. Not many authors make as much of an effort as they do to condense their message and reward the reader for taking the time to read an actual book. It’s insulting how much work some authors expect you to put in before they will reveal their secrets!

Legacy code

I’m happy that I’ve discovered the work of P.M. Jones recently. His and mine interests seem to align at several interesting points. Though I don’t personally enjoy “putting .308 holes in targets at 400 yards” (as quoted by Phil Sturgeon), I do care a great deal about package coupling (and cohesion for that matter) and I’m also lightly inflammable when it comes to the use of service locators. It also appears that Paul, just like myself and many others in this business, has felt the pain of working on a legacy PHP application, trying to add features to it, or change existing behavior. This is a particularly hard thing to do and because so many incompetent developers have been creating PHP applications since the dawn of PHP, chances are that a big part of the job of competent PHP developers nowadays consists of maintaining these dumpsites of include statements and superglobals.

During a recent run of my Advanced Web Application Architecture training, we discussed the distinction between infrastructure code and non-infrastructure code, which I usually call core code. One of the participants summarized the difference between the two as: “everything in your vendor directory is infrastructure code”. I don’t agree with that, and I will explain why in this article.

Not all code in vendor is infrastructure code

Admittedly, it’s easy for anyone to not agree with a statement like this, because you can simply make up your own definitions of “infrastructure” that turn the statement false. As a matter of fact, I’m currently working on my next book (which has the same title as the training), and I’m working on a memorable definition that covers all the cases. I’ll share with you the current version of that definition, which consists of two rules defining core code. Any code that doesn’t follow both these rules at the same time, should be considered infrastructure code.

Here are some rules I use for working with dynamic arrays. It’s pretty much a Style Guide for Array Design, but it didn’t feel right to add it to the Object Design Style Guide, because not every object-oriented language has dynamic arrays. The examples in this post are written in PHP, because PHP is pretty much Java (which might be familiar), but with dynamic arrays instead of built-in collection classes and interfaces.

Here are some rules I use for working with dynamic arrays. It’s pretty much a Style Guide for Array Design, but it didn’t feel right to add it to the Object Design Style Guide, because not every object-oriented language has dynamic arrays. The examples in this post are written in PHP, because PHP is pretty much Java (which might be familiar), but with dynamic arrays instead of built-in collection classes and interfaces.

Here are some rules I use for working with dynamic arrays. It’s pretty much a Style Guide for Array Design, but it didn’t feel right to add it to the Object Design Style Guide, because not every object-oriented language has dynamic arrays. The examples in this post are written in PHP, because PHP is pretty much Java (which might be familiar), but with dynamic arrays instead of built-in collection classes and interfaces.

Today Manning released my latest book! It’s called “Object Design Style Guide”.

In November 2018 I started working on this book. The idea for it came from a conversation I had with the friendly folks at Akeneo (Nantes) earlier that year. It turned out that, after days of high level training on web application architecture and Domain-Driven Design, there was a need for some kind of manual for low level object-oriented programming. Not as low level as the kind of programming advice people usually refer to as clean code, but general programming rules for different kinds of objects. For instance:

Today Manning released my latest book! It’s called “Object Design Style Guide”.

In November 2018 I started working on this book. The idea for it came from a conversation I had with the friendly folks at Akeneo (Nantes) earlier that year. It turned out that, after days of high level training on web application architecture and Domain-Driven Design, there was a need for some kind of manual for low level object-oriented programming. Not as low level as the kind of programming advice people usually refer to as clean code, but general programming rules for different kinds of objects. For instance:

Please note: The Object Design Style Guide is now available via Manning,

With a foreword by Ross Tuck

Today I’ve released the revised edition of my latest book “Style Guide for Object Design”. Over the past few weeks I’ve fixed many issues, smaller and larger ones. Ross Tuck and Laura Cody have read the manuscript and provided me with excellent feedback, which has helped me a lot to get the book in a better shape. I added many sections, asides, and even an extra chapter, because some topics deserve more detail and background than just a few paragraphs. Oh, and Ross wrote the kindest of forewords to kick off the book. It’s available in the free sample of the book.

All of a sudden it became book-writing season. It began in August when I started revising my second book, “Principles of Package Design”. Apress had contacted me about adopting it, and they didn’t want to change a lot about it. However, the book was from 2015 and although I had aimed for it to be “timeless”, some parts needed an update. Furthermore, I had happily pressed the “Release” button back then, but it’s the same as with software development: the code you wrote last year, you wouldn’t approve of today.

During the great PHP Benelux Conference 2015 edition I published the final version of Principles of Package Design. Let me tell you a little bit about its history and some of the things I had to deal with.

A bit of history

My first book, A Year With Symfony, was released on September 4th, 2013. Just 6 days later, I started working on Principles of PHP Package Design. As it turned out, it was quite hard to keep the process going. There were several months in which I wrote nothing at all. Sometimes I picked up the work, but then I had completely lost my track and tried to get back on it by revising existing chapters over and over again. Meanwhile all kinds of other projects begged for attention as well, including the release of the backend project of the new nos.nl website, the preparations for the Hexagonal architecture training and the birth of our daughter Julia ;)

Seven months, two presentations and three blog posts later, I’ve published the first installment of my new book, Principles of PHP Package Design.

From the introduction of the book:

Naturally the biggest part of this book covers package design principles. But before we come to that, we first take a close look at what constitutes packages: classes and interfaces. The way you design them has great consequences for the characteristics of the package in which they will eventually reside. So before considering package design principles themselves, we first need to take a look at the principles that govern class design. These are the so-called SOLID principles. Each letter of this acronym stands for a different principle, each of which we will briefly (re)visit in the first part of this book.

My first book, A Year With Symfony, has been available since September 4th last year. That’s not even six months ago, but right now it has almost 800 readers already (digital and printed edition combined).

During the past few days I’ve taken the time to write an extra chapter for the book. Consider it as a big thank you to everybody who bought the book! I feel very much supported by the Symfony/PHP community and this really keeps me going.

A couple of weeks ago the reader count of my first book A Year With Symfony reached the number 365. It seemed to me an appropriate moment to write something about my experiences. In this post, I will not be too humble, and just cite some people who wrote some very nice things about my book on Twitter.

The book was highly anticipated ever since I first wrote something about it on Twitter. Leanpub book pages have a nice feature where users can subscribe to news about the book release.

After working on my book “A Year With Symfony” for four months, I published it yesterday at the Dutch Symfony Usergroup meetup in Amsterdam. It was really great to be able to do this real-time while everybody in the room was looking at the screen waiting for the build process to finish.

The book is hosted on Leanpub, which is a great platform for writing and publishing books. They provide the entire infrastructure. You get a good-looking landing page and they handle all things related to payment.

This blog runs on Sculpin, a static site generator. The generator itself runs on Symfony, which for me makes it easy to extend. However, I find that if you want something special, it can usually be done, but it may take several hours to get it right. In the end though, the solution is often quite elegant.

A custom content type for events

One custom feature I wanted for this website was a list of events (conference talks, trainings, etc.). Sculpin’s documentation suggests using a custom content type for that. This allows you to create a directory with files, each of which will be considered an “event”.

Whenever I run the Sculpin generate command to generate a new version of the static website that is this blog, I notice there are a lot of useless files that get copied from the project’s source/ directory to the project’s output/ directory. All the files in the output/ directory will eventually get copied into a Docker image based on nginx (see also my blog series on Containerizing a static website with Docker). And since I’m on a hotel wifi now, I realized that now was the time to shave off any unnecessary weight from this Docker image.

In the previous posts we looked at creating a build container, and after that we created a blog container, serving our generated static website.

It’s quite surprising to me how simple the current setup is — admittedly, it’s a simple application too. It takes about 50 lines of configuration to get everything up and running.

The idea of the blog container, which has nginx as its main process, is to deploy it to a production server whenever we feel like it, in just “one click”. There should be no need to configure a server to host our website, and it should not be necessary to build the application on the server too. This is in fact the promise, and the true power of Docker.

In the previous post we looked at the process of designing a build container, consisting of all the required build tools for generating a static website from source files. In order to see the result of the build process, we still need to design another container, which runs a simple web server, serving the static website (mainly .html, .css, .js and .jpg files).

Designing the blog container

We’ll use a light-weight install of Nginx as the base image and simply copy the website files to the default document root (/usr/share/nginx/html) (only after removing any placeholder files that are currently inside that directory). The complete file docker/blog/Dockerfile looks like this:

Recently a former colleague of mine, Lucas van Lierop, showed me his new website, which he created using Spress. Lucas took two bold moves: he started freelancing, and he open-sourced his website code. This to me was very inspiring. I’ve been getting up to speed with Docker recently and am planning to do a lot more with it over the coming months, and being able to take a look at the source code of up-to-date projects that use Docker is certainly invaluable.

Previously we looked at how to use the static site generator Sculpin to generate in-project documentation. When Sculpin builds the HTML files and assets it looks at what files are in the source/ directory and processes them based on certain rules (e.g. “parse Markdown files”, “format HTML files with Twig”, etc.). The purpose for my “living documentation” project is to also dynamically generate documentation based on PHP and configuration files from the main project. For example, consider extracting a Composer dependency diagram. I don’t want to manually copy the diagram and an HTML template to the source/ directory. I want Sculpin to generate a current diagram on-demand and wrap it inside a template.

Recently I’ve been reading the book Living documentation by Cyrille Martraire. I can warmly recommend this book to you. It basically provides a solution for the age-old problem of creating and maintaining accurate, up-to-date and useful project documentation. One of the key ideas is to generate documentation instead of writing it. This should help prevent duplication and outdated information that is not trust-worthy and would therefore be neglected. I’m currently looking for ways to technically accomplish such a thing with PHP projects. This should result in reusable tools which will make it easier and more fun to document future projects while writing the code.

I wanted to be able to deploy MailComments to my Digital Ocean droplet (VPS) easily and without thinking. Due to a lack of maintenance, some more “operations” work had piled up as well:

• The Digital Ocean monitoring agent had to be upgraded, but apt didn’t have enough memory to do that on this old, small droplet.
• The Ubuntu version running on that droplet was also a bit old by now.
• The easiest thing to do was to just create a new droplet and prepare it for deploying my personal websites.
• Unfortunately, my DNS setup was completely tied to the IP address of the droplet, so I couldn’t really create a new droplet, and quickly switch. I’d have to wait for the new DNS information to propagate.

These issues were in the way of progress, so I decided to take some more time to rearrange things.

I wanted to be able to deploy MailComments to my Digital Ocean droplet (VPS) easily and without thinking. Due to a lack of maintenance, some more “operations” work had piled up as well:

• The Digital Ocean monitoring agent had to be upgraded, but apt didn’t have enough memory to do that on this old, small droplet.
• The Ubuntu version running on that droplet was also a bit old by now.
• The easiest thing to do was to just create a new droplet and prepare it for deploying my personal websites.
• Unfortunately, my DNS setup was completely tied to the IP address of the droplet, so I couldn’t really create a new droplet, and quickly switch. I’d have to wait for the new DNS information to propagate.

These issues were in the way of progress, so I decided to take some more time to rearrange things.

For my new workshop - “Building Autonomous Services” - I needed to define several Docker containers/services with more or less the same setup:

1. A PHP-FPM process for running the service’s PHP code.
2. An Nginx process for serving static and dynamic requests (using the PHP-FPM process as backend).

To route requests properly, every Nginx service would have its own hostvcname. I didn’t want to do complicated things with ports though - the Nginx services should all listen to port 80. However, on the host machine, only one service can listen on port 80. This is where reverse HTTP proxy Traefik did a good job: it is the only service listening on the host on port 80, and it forwards requests to the right service based on the host name from the request.

Here’s what I wanted to do:

• Run the official redis image as a service in a cluster set up with Docker Swarm.
• Configure a password to be used by connecting clients. See also Redis’s AUTH command. The relevant command-line option when starting the redis server would be --requirepass.

This is just a quick post, sharing what I’ve figured out while trying to accomplish all of this. I hope it’s useful in case you’re looking for a way to make a container image (an official one or your own) ready to be used with Docker Secrets.

The “builder pattern”

As a programmer you may know the Gang-of-Four Builder design pattern. The Docker builder pattern has nothing to do with that. The builder pattern describes the setup that many people use to build a container. It involves two Docker images:

1. a “build” image with all the build tools installed, capable of creating production-ready application files.
2. a “service” image capable of running the application.

Sharing files between containers

At some point the production-ready application files need to be copied from the build container to the host machine. There are several options for that.

In the previous posts we looked at creating a build container, and after that we created a blog container, serving our generated static website.

It’s quite surprising to me how simple the current setup is — admittedly, it’s a simple application too. It takes about 50 lines of configuration to get everything up and running.

The idea of the blog container, which has nginx as its main process, is to deploy it to a production server whenever we feel like it, in just “one click”. There should be no need to configure a server to host our website, and it should not be necessary to build the application on the server too. This is in fact the promise, and the true power of Docker.

In the previous post we looked at the process of designing a build container, consisting of all the required build tools for generating a static website from source files. In order to see the result of the build process, we still need to design another container, which runs a simple web server, serving the static website (mainly .html, .css, .js and .jpg files).

Designing the blog container

We’ll use a light-weight install of Nginx as the base image and simply copy the website files to the default document root (/usr/share/nginx/html) (only after removing any placeholder files that are currently inside that directory). The complete file docker/blog/Dockerfile looks like this: