Technical Debt

Wikipedia defines technical debt as "neologistic metaphor referring to the eventual consequences of slapdash software architecture and hasty software development." In general, I consider technical debt as all the things in software development ( architectural design, refactoring, unit testing ), that are normally considered peripheral to the development of the main business functionality. All these things are needed to ensure long term quality of the software product.

It is not unusual to see a tug-of-war between product management and engineering about how much can be done in a given period of time. Engineering teams often find it hard to communicate the technical debt that they have to pay for the required changes. Try explaining unit testing, automated testing, refactoring and other such forms of technical debt to business users. But these things are very important for the long term health of the critical software systems. So the technical debt affects the business in the long run.

Often the technical debt accumulates and has to be paid by drastic measures such as stopping all future releases till the required clean up of the code base has been done. I think the term "debt" in technical debt is just awesome. Just like financial debt, if the periodic payments of technical debt are not made, serious consequences ensue. So how can the product managers avoid the painful drastic measures that are almost guaranteed, unless the technical debt is paid on a regular basis ?

Marty Cagan offers a simple suggestion in his book "Inspired". He recommends product managers to allow the engineering teams to use up to 20% of their capacity for maintenance or any other technical tasks that the engineers deem required to ensure long term health of the software systems. Marty says: "They ( engineers ) might use it to rewrite, re-architect or re-factor problematic parts of the code base, or to swap out database management systems, improve system performance - whatever they believe is necessary to avoid ever having to come to the team and say, 'we need to stop and rewrite'. If you are in really bad shape today, you might need to make this 30% or even more of the resources ".

I like this recommendation a lot. But based on my experience I think it can be very hard to get that magic 20% capacity on a regular basis, considering the constant urgency to deliver business functionality faced by teams.

Marty explains that many companies, that don't pay the technical debt on a regular basis, often face existential crises. He gives the example of eBay, Friendster and Netscape. At some point, engineering teams of these companies told the product managers that they can't deliver any more features and that they have to stop and rewrite the whole code base which was in a mess. eBay came to near collapse because of such situation in 1999. Friendster and Netscape were never able to recover.

Bottom line is, it is in the long term business interest of a software driven company ( which most companies are these days ), to pay the technical debt on a regular basis. Allocating 20% of the engineering capacity to do that, is a very reasonable recommendation.