Too much magic?

Posted on by Matthias Noback

Years ago my co-worker Maurits introduced me to the term "magic" in programming. He also provided the valuable dichotomy of convention and configuration (or in fact, he'd choose configuration over convention...). I think this distinction could be very helpful in psychological research, figuring out why some people prefer framework X over framework Y. One requires the developer to spell out everything they want in elaborate configuration files, the other relies on convention: placing certain files with certain names and certain methods in certain places will make everything work "magically".

And there we are: magic. Often used in code reviews and discussions: "there's too much magic here". Yesterday the word popped up in a Twitter thread as well:

"symfony has too much magic, to its own detriment..." @bazinder

This was answered with:

"I'd say that everything is magic until you start to understand it :D" @iosifch

It made me wonder, what should we consider to be "magic" in programming? Is magic in code okay, or should it be avoided at all cost?

As an example of magic, the fact that you can define a controller like this, is already magical:

/**
 * @Route("/task")
 */
final class TaskController
{
    /**
     * @Route("/new")
     */
    public function new(Request $request): Response
    {
        // ...
    }
}

Who invokes it? Why, and when? You can't figure that out by clicking "Find usages..." in PhpStorm!

This innocent example shows how quick we are to accept magic from a framework. Just do things "the framework way", put this file there, add these annotations, and it'll work. As an alternative, we could set up an HTTP kernel that doesn't need any magic. For instance, we could write the dispatching logic ourselves:

$routes = [
    '#^/task/new$#' => function (Request $request): Response {
        $controller = new TaskController();
        return $controller->new($request);
    },
    // ...
];

foreach ($routes as $path => $dispatch) {
    if (/* request URI matches path regex */) {
        $response = $dispatch($request);
        // Render $response to the client
        exit();
    }
}

// Show 404 page

Of course we wouldn't or shouldn't do this, but if we did, at least we'd be able to pinpoint the place where the controller is invoked, and we'd be able to inject the right dependencies as constructor arguments, or pass additional method arguments. Of course, a framework saves us from writing all these lines. It takes over the instantiation logic for the controller, and analyzes annotations to build up something similar to that $routes array. It allows other services to do work before the controller is invoked, or to post-process the Response object before it's rendered to the client.

The more a framework is going to do before or after the controller is invoked, the more magical the framework will be. That's because of all the dynamic programming that's involved when you make things generic. E.g. you can add your own event subscribers that modify the Request or Response, or even by-pass the controller completely. It's unclear if and when such an event subscriber will be invoked, because it happens in a dynamic way, by looping over a list of event subscriber services. If you have ever step-debugged your way from index.php to the controller, you know that you'll encounter a lot of abstract code, that is hard to relate to. It's even hard to figure what exactly happens there.

I'm afraid there's no way around magic. If you want to use a framework, then you import magic into your project. Circling back to Iosif's comment ("everything is magic until you start to understand it"), I agree that the way to deal with your framework's magic is to understand it, know how everything works under the hood. It doesn't make the magic go away, but at least you know how the trick works. Personally I don't think this justifies relying on all the magic a framework has to offer. I think developers should need as little information as possible to go ahead and change any piece of code. If they want to learn more about it,

  • They should be able to "click" on method calls, to zoom in on what happens behind the call.
  • They should be able to click on "Find usages" to zoom out and figure out how and where a method is used.

When you get to the magical part of your code base, usually the part that integrates with the framework or the ORM, then none of this is possible. You can't click on anything, you just have to "know" how things work. I think this is a maintainability risk. If you don't know how a piece of code works, you're more likely to make mistakes, and it becomes less and less likely that you'll even dare to touch it. Which is why I prefer more explicit, less magical code, that is safer to change because every aspect is in plain sight. When it comes to framework integration code, we can never make everything explicit, or we should rather dump the framework entirely. So how can we find some middle ground; how can we find a good balance between framework magic and explicit, easy to understand and change code? There are three options:

  1. When frameworks offer an explicit and a magical option for some feature, use the more explicit alternative.
  2. Replace magical features with your own, hand-written, and more explicit alternative.
  3. Keep using the magical feature, but document it.

As an example of 1: I don't want models/entities to be passed as controller arguments.

/**
 * @Route("/edit/{id}")
 */
public function edit(Task $task, Request $request): Response
{
    // ...
}

Instead, I want to see in the code where this object comes from, and based on what part of the request:

final class TaskController
{
    public function __construct(
        private TaskRepository $taskRepository
    ) {
    }

    public function edit(Request $request): Response
    {
        $task = $this->taskRepository
            ->getById($request->attributes->getInt('id'));

        // ...
    }

Another example of 1: if I can choose between accessing a service in a global, static way (e.g. using a fa├žade) or as a constructor-injected dependency, I choose the latter, which is the less magical one.

As an example of 2: instead of letting Doctrine save/flush my entity to the database, including any other entity it has loaded, I often explicitly map the data of one entity, so I can do an UPDATE or INSERT query myself (see my article about ORMless).

As an example of 3: when defining routes, or column mappings, I do it close to where the developer is already looking. I use a @Route annotation (or attribute) instead of defining it in a .yml file that lives in a completely different place. If I still like to let Doctrine map my entities, I make sure to have the @Column annotations next to the entity's properties, instead of in a separate file. If a developer needs to change something, it will be easier to understand what's going on and what else needs to be changed. The annotations serve as a reminder of the magic that's going on.

By using these tactics I think you can get rid of a lot of code that relies on some of the framework's most over-the-top magic. If we manage to do that, we can spend less time learning about the inner workings of framework X. We'll have fewer questions on StackOverflow that are like "How can I do ... with framework Y?" If we apply these tactics, there will be fewer differences between code written for framework X or framework Y. It means the choice for a particular framework becomes less relevant. All of the framework-specific knowledge doesn't have to be preserved; new team members don't have to be "framework" developers. If they can accept a request and return a response, they should be fine. Job ads no longer have to mention any framework. Developers don't have to do certification exams, watch video courses, read books, and so on. They can just start coding from day 1.

PHP Symfony Laravel decoupling
Comments
This website uses MailComments: you can send your comments to this post by email. Read more about MailComments, including suggestions for writing your comments (in HTML or Markdown).
Ben Scheffer
Hi Matthias,

First of all: Thanks for writing understandable posts about interesting subjects. For a self-thaught full-stack developer with nearly five years of job experience like me, it's great to be able to learn from someone with this much knowledge and expertise.

Then about the magic in frameworks:
I'll be 59 nine in a few months and started to properly learn php only in 2017. I say 'properly' because as 'webdesigner' I started building sites in the early 2000's, but those where drupal and wordpress sites. I kind of avoided getting my hands dirty by digging in the code. Btw. speaking of magic: for a content creator, cms's are very magical to.
Last week I started learning Symfony. Not that I felt the urge but because it will become part of my job soon. And that's my two cents for your well explained statement about avoiding framework trickery (can I say it like that?): there seems to be a demand for symfony, laravel - and to lesser extend - Zend/Laminas developers. The same thing goes for the front-end. I rarely see job openings for JavaScript developers. It's almost always for React, Vue or Angular. Now while I believe that a developer should have an extensive knowledge about the programming language they're working with, when they want (or need) to build something with a framework, I'd say, they better do it the framework way. Maybe this is why frameworks exist. They enable jumiors and mediors like me to start building applications fast. Otherwise web agencies could only hire seniors. Don't get me wrong though: I hope to become a senior (php) developer like you well before I become a senior, if you catch my pun.

Thanks again.

Ben (from Friesland btw.)

Matthias Noback

Thanks, Ben, and a very good point. In my experience it really helps to know how to work with a framework and to get things done with it. Therefore, when I talk about getting rid of or never even using some magic parts of a framework, I believe it's something that works best when you have more experience. Not just with programming, but also with old projects. That's when you recognize that certain approaches will quickly get out of hand and become unmaintainable in the long run. Personally I'm always looking for a pragmatic approach though, since you may not always run into those problems anyway. I'd like to invest ~20% extra development time, to gain ~80% of the advantages from decoupling. I'm writing about this in my upcoming book "Recipes for decoupling", maybe that will be interesting for you when it is published (which will happen within a few months).

Good luck with your career as a programmer!