Too much magic?

Years ago my co-worker Maurits introduced me to the term “magic” in programming. He also provided the valuable dichotomy of convention and configuration (or in fact, he’d choose configuration over convention…). I think this distinction could be very helpful in psychological research, figuring out why some people prefer framework X over framework Y. One requires the developer to spell out everything they want in elaborate configuration files, the other relies on convention: placing certain files with certain names and certain methods in certain places will make everything work “magically”.

And there we are: magic. Often used in code reviews and discussions: “there’s too much magic here”. Yesterday the word popped up in a Twitter thread as well:

“symfony has too much magic, to its own detriment…” @bazinder

This was answered with:

“I’d say that everything is magic until you start to understand it :D” @iosifch

It made me wonder, what should we consider to be “magic” in programming? Is magic in code okay, or should it be avoided at all cost?

As an example of magic, the fact that you can define a controller like this, is already magical:

/**
 * @Route("/task")
 */
final class TaskController
{
    /**
     * @Route("/new")
     */
    public function new(Request $request): Response
    {
        // ...
    }
}

Who invokes it? Why, and when? You can’t figure that out by clicking “Find usages…” in PhpStorm!

This innocent example shows how quick we are to accept magic from a framework. Just do things “the framework way”, put this file there, add these annotations, and it’ll work. As an alternative, we could set up an HTTP kernel that doesn’t need any magic. For instance, we could write the dispatching logic ourselves:

$routes = [
    '#^/task/new$#' => function (Request $request): Response {
        $controller = new TaskController();
        return $controller->new($request);
    },
    // ...
];

foreach ($routes as $path => $dispatch) {
    if (/* request URI matches path regex */) {
        $response = $dispatch($request);
        // Render $response to the client
        exit();
    }
}

// Show 404 page

Of course we wouldn’t or shouldn’t do this, but if we did, at least we’d be able to pinpoint the place where the controller is invoked, and we’d be able to inject the right dependencies as constructor arguments, or pass additional method arguments. Of course, a framework saves us from writing all these lines. It takes over the instantiation logic for the controller, and analyzes annotations to build up something similar to that $routes array. It allows other services to do work before the controller is invoked, or to post-process the Response object before it’s rendered to the client.

The more a framework is going to do before or after the controller is invoked, the more magical the framework will be. That’s because of all the dynamic programming that’s involved when you make things generic. E.g. you can add your own event subscribers that modify the Request or Response, or even by-pass the controller completely. It’s unclear if and when such an event subscriber will be invoked, because it happens in a dynamic way, by looping over a list of event subscriber services. If you have ever step-debugged your way from index.php to the controller, you know that you’ll encounter a lot of abstract code, that is hard to relate to. It’s even hard to figure what exactly happens there.

I’m afraid there’s no way around magic. If you want to use a framework, then you import magic into your project. Circling back to Iosif’s comment (“everything is magic until you start to understand it”), I agree that the way to deal with your framework’s magic is to understand it, know how everything works under the hood. It doesn’t make the magic go away, but at least you know how the trick works. Personally I don’t think this justifies relying on all the magic a framework has to offer. I think developers should need as little information as possible to go ahead and change any piece of code. If they want to learn more about it,

• They should be able to “click” on method calls, to zoom in on what happens behind the call.
• They should be able to click on “Find usages” to zoom out and figure out how and where a method is used.

When you get to the magical part of your code base, usually the part that integrates with the framework or the ORM, then none of this is possible. You can’t click on anything, you just have to “know” how things work. I think this is a maintainability risk. If you don’t know how a piece of code works, you’re more likely to make mistakes, and it becomes less and less likely that you’ll even dare to touch it. Which is why I prefer more explicit, less magical code, that is safer to change because every aspect is in plain sight. When it comes to framework integration code, we can never make everything explicit, or we should rather dump the framework entirely. So how can we find some middle ground; how can we find a good balance between framework magic and explicit, easy to understand and change code? There are three options:

1. When frameworks offer an explicit and a magical option for some feature, use the more explicit alternative.
2. Replace magical features with your own, hand-written, and more explicit alternative.
3. Keep using the magical feature, but document it.

As an example of 1: I don’t want models/entities to be passed as controller arguments.

/**
 * @Route("/edit/{id}")
 */
public function edit(Task $task, Request $request): Response
{
    // ...
}

Instead, I want to see in the code where this object comes from, and based on what part of the request:

final class TaskController
{
    public function __construct(
        private TaskRepository $taskRepository
    ) {
    }
    
    public function edit(Request $request): Response
    {
        $task = $this->taskRepository
            ->getById($request->attributes->getInt('id'));
    
        // ...
    }

Another example of 1: if I can choose between accessing a service in a global, static way (e.g. using a façade) or as a constructor-injected dependency, I choose the latter, which is the less magical one.

As an example of 2: instead of letting Doctrine save/flush my entity to the database, including any other entity it has loaded, I often explicitly map the data of one entity, so I can do an UPDATE or INSERT query myself (see my article about ORMless).

As an example of 3: when defining routes, or column mappings, I do it close to where the developer is already looking. I use a @Route annotation (or attribute) instead of defining it in a .yml file that lives in a completely different place. If I still like to let Doctrine map my entities, I make sure to have the @Column annotations next to the entity’s properties, instead of in a separate file. If a developer needs to change something, it will be easier to understand what’s going on and what else needs to be changed. The annotations serve as a reminder of the magic that’s going on.

By using these tactics I think you can get rid of a lot of code that relies on some of the framework’s most over-the-top magic. If we manage to do that, we can spend less time learning about the inner workings of framework X. We’ll have fewer questions on StackOverflow that are like “How can I do … with framework Y?” If we apply these tactics, there will be fewer differences between code written for framework X or framework Y. It means the choice for a particular framework becomes less relevant. All of the framework-specific knowledge doesn’t have to be preserved; new team members don’t have to be “framework” developers. If they can accept a request and return a response, they should be fine. Job ads no longer have to mention any framework. Developers don’t have to do certification exams, watch video courses, read books, and so on. They can just start coding from day 1.

• Symfony
• Laravel
• Decoupling