Dealing with technical debt during the sprint

Matthias Noback

It’s quite ironic that my most “popular” tweet has been posted while Twitter itself is in such a chaotic phase. It’s also quite ironic that I try to provide helpful suggestions for doing a better job as a programmer, yet such a bitter tweet ends up to be so popular.

Twitter and Mastodon are micro-blogging platforms. The problem with micro-blogs, and with short interactions in general, is that everybody can proceed to project onto your words whatever they like. So at some point I often feel the need to explain myself with more words, in an “actual” blog like this one.

Waste in scrum processes

Hypothesis: the moment a team adds the requirement that each PR/commit should be related to a Jira issue, it will start accumulating even more tech debt than before.

We notice that one of the dependencies of our project has been marked as “abandoned” and we need to upgrade/switch to something else. When helping a new co-worker join the team, we find out that some crucial steps aren’t documented in the README. The test framework hasn’t been updated for some time, and upgrading means we have to rewrite some test setup code.

These are things that just happen to our projects from time to time. Many developers won’t go ahead and make the necessary changes. Being part of a “scrum process” they will:

  • Create an issue in the backlog
  • Bring the issue to the attention of the decision maker
  • Wait for the decision maker to assign it to a future sprint

Finally, once the issue has been assigned to a sprint, it needs to be refined. So a group of people that is often too large starts to talk about and describe what needs to be done. Why? Because scrum tradition prescribes that every person on the team should be able to pick up the issue. I don’t think that’s true at all; we’re just wasting time explaining everything to everyone, while often only 2 people are going to pick it up.

In the end we have to vote for the number of story points that we’re going to assign. This magic number has no specific meaning. If we try to describe what such a point represents, we get different answers, even within the same team:

  1. “It’s definitely not how much time we spend on it” (because we don’t want it to be an estimate)
  2. “It’s how complex we think this is” (unfortunately we all have different ideas about what “complexity 3” means)
  3. “We can do about 50 points of these in each sprint” (so it is an estimate after all; given we have two weeks, we can do 50 of them, so each point represents 2 weeks divided by 50 of our time)

Don’t get me wrong, it’s good to think about how much time something will likely take and use a rough estimate to decide if you want to start working on it. It’s just that we are guessing, and we can have huge surprises while we are actually doing the work. Or the opposite happens: we are over-complicating things in the refinement stage and it turns out the actual work was so easy, we get 5 points for the price of 1…

In both cases, a big part of the preparation phase is just a waste of time and energy. I’m certain that many scrum teams could do much, much more if they would let go of wasteful practices like those that “official scrum” or similar project management techniques prescribe.

How to deal with technical debt in a sprint

Back to the original point about technical debt. From time to time developers will notice something about the code base that really needs to be improved, something that is not part of any feature anybody is working on, just “general maintenance” or “developer experience”, and so on. Developers need to do this work because they have to battle the forces pulling the project downward. If they do not continuously do that, one day the project will be beyond repair. Yet, the scrum process prescribes that no work be done in the sprint that is not on the board. So an issue has to be created, and we jump back into the project management waterfall.

At this point a developer have several options, each of which appeared several times in the comments to my bitter tweet:

  • They can follow the process because they have to. There are strict requirements, maybe an ISO-standard, that have to be enforced. No way around it. So first they create an issue, and wait for it to be assigned to a sprint.
  • In some teams they wouldn’t have to wait, they can pick it up inside the sprint. But only if it’s small, “a 1” or “a 2”, because more would “endanger” the sprint.
  • They can play the process, and do general improvements as part of the current ticket they are working on (but these are now unrelated changes).
  • They can commit their changes to a dedicated Technical Debt ticket (but the issue itself is quite meaningless).
  • They can be passive-aggressive and commit their changes to a random ticket.

I can’t speak for everyone and every team, but in my experience, developers are even less likely to improve structural issues (technical debt) if they have to deal with this slow and demotivating process. They don’t create those tech debt issues anymore, and the project is going to decline faster.

That’s really sad, because most developers I’ve met are well able to do what’s good for the project. They know what will help the project survive. Yet the process keeps them from doing it. So I propose just skipping the whole scrum waterfall in these cases.

Objections

Of course, not everything should go “under the radar”. We need planning, discussions, exchanging ideas. We need to challenge solutions, adopt a domain-oriented collaboration style, and so on. But, there are also things we just have to do as developers, and if you ask me, we shouldn’t always go through the process. These things just need to be done, sooner rather than later. So many projects have already accumulated so much technical debt that now is the time to act.

Some common objections I’ve heard to skipping the whole scrum process for technical debt-resolving tickets (rephrasing here):

  1. Developers may no longer spend time on “real work”, they become renegades, improving code forever, never delivering any value for the business or the users.
  2. Developers may make mistakes, in judgement and code. They should still create an issue.

These objections make sense to me personally:

  1. I once was such a renegade developer who didn’t care about the user, the business, only about cool code and the cool framework. Yes, you need some way to make sure that the work these developers do is visible, and relevant. I’m not sure if scrum is the best option.
  2. We should help everyone not to make mistakes, choose the right solutions, and so on. But… creating an issue isn’t saving us from those mistakes. Only if somebody thinks about the problem, and talks with you about the right solution. Which means it’s not about the issue per se, you just need to collaborate. For this, I recommend pair or team programming, and building a good test suite.

Conclusion

In short, I think most developers know what’s right, what needs to be done to keep the project in a good shape. Just let them do it. But not alone, and not in the shadows.