Managing technical debt is one of the most critical tasks in digital program management. In addition, it can determine sustained success or eventual failure.

Recently, authors in software development and DevOps have stressed that digital programs must allocate a fixed amount of development bandwidth to pay down technical debt, or else they will eventually suffer the consequences. In the day to day blocking & tackling of project management, it’s easy to feel like we don’t have enough time to worry about the future. However, we should really tell ourselves that we don’t have the luxury of not worrying about technical debt.

So why is this so important?

Technical debt and the art of slowing down to speed up

Several years ago, I led a team to design a Customer 360 portal for a global heavy equipment manufacturer that would transform the company’s approach to interacting with both its network of dealerships and its end customers. We planned to roll this portal out in 15 languages and 140 countries.

After a slow initial start, the program had hit its stride within a few months. With development teams pushing code with increasing velocity, the sales & marketing teams grew excited about their new product. The first few pilot releases were met with fanfare. Everything was going well.

However, only six months later, the tone was markedly more somber. Development progress had slowed. With each new code release, more defects appeared. At one point, it seemed like fixing one defect caused three more. After that, the client became nervous. Then they became furious at us.

Only by taking the painful step of halting all new development and fixing problems that had lingered for months could our team regain our composure. That was the easy part. After that, we never regained the client’s trust. In the end, they were happy we were finished, but they were also glad to be rid of us.

Why did our Customer 360 project go so right in the beginning and end so poorly? I struggled at first to understand where it all went wrong.

In short, we failed to account for technical debt along the way. Instead, we needed to slow down in order to speed up.

What is technical debt?

Technical debt consists mostly of the invisible issues inherent in software development. It can be either inadvertent or deliberate.

Ward Cunningham introduced the term in 1992 as “not-quite-right” code that we postpone making right¹.

Martin Fowler’s classic definition divides technical debt into a 2×2 matrix².

Broadly, technical debt can involve architecture, development, testing, or even documentation. It impacts both the evolution and maintainability of the end product. Technical debt is mostly invisible, issues that lay hidden within the inner workings of a software product.

For example, consider the following technical debt landscape³.

While our attention naturally gravitates toward the visible, external issues, it is more important to take into account these mostly invisible issues.

The impact of technical debt

Gene Kim, Patrick Debois, John Willis, and Jez Humble provide an alarming analysis. In their seminal book on DevOps, they estimated that the opportunity cost of improper IT management is $2.6 trillion per year⁴:

• Approximately $520 billion of IT spend is wasted on both planned and unplanned rework each year
• Diverting just half of that waste into higher value activities could create a 5x gain (a conservative estimate), for $2.6 trillion of value per year
• Furthermore, this estimate does not even factor in the human cost of long hours, weekend work, and decreased quality of life

In comparison, the annual GDP of the United Kingdom is also $2.6 trillion⁵.

In other words, the opportunity for managing technical debt is comparable to the GDP of the United Kingdom.

Technical debt and the three vicious circles

The primary reason why managing technical debt is so important is because the cost of repaying the debt can easily spiral out of control. Like financial debt, the longer we wait, the more expensive it becomes to pay.

There are three types of vicious circles⁶: contagious debt, hidden debt, and lack of uniformity and familiarity.

Contagious debt

Suppose on a software project, a database (component A) did not have a standard interface. At the time, the team justified this decision, because they wanted to create a “better” version in the future.

However, that means that component B needs to access A directly rather than via a standard interface. In other words, the debt spreads from B to A.

Later on, when the company planned to create specialized components C and D that also interfaced with A, they realized they could no longer delay the standard interface for A.

However, upon investigating the technical requirements, the teams realize that the cost of refactoring A now includes the cost of refactoring B. Because they waited to pay down A’s technical debt, that debt became more expensive as time passed.

Hidden debt

Hidden debt refers to the cost of all the unknowns involved in software development. Behind contagious debt, it is the second most vicious circle.

For example, suppose that over the course of a software product, one team had created three different communication protocols between different components. Recognizing that this would be unsustainable over the long term, the team made the conscientious decision to refactor all three protocols into one common, standardized protocol.

However, ripple effects of hidden debt led to incorrect estimate of refactoring effort. When the team began to analyze the effort involved in replacing the three different components, they quickly uncovered a set of previous unknowns.

Eventually , due to time pressure, the team ended up creating a fourth protocol, without having removed the other three. In the end, their conscientious decision to pay down technical debt ironically led to the creation of more technical debt.

Lack of uniformity and familiarity

The third type of vicious circle refers to the people element of technical debt. We need to balance uniformity of policies and design patterns with the creativity and freedom that developers need in order to solve their daily challenges.

Combined with the Agile practice of having teams modifying parts of the system they are not familiar with, lack of uniformity and familiarity can over time lead to lower software quality and less robust codebases.

The psychology of technical debt

Given the costs of technical debt, we may wonder why managing technical debt is so difficult. Why don’t digital programs just pay down technical debt up front?

In truth, human psychology makes managing technical debt difficult. We are cognitively limited by two biases: loss aversion bias and concreteness bias.

Loss aversion bias

Loss aversion refers to the human tendency to prefer avoiding losses over acquiring gains. For example, suppose we had a choice between:

• A certain loss of $100
• 50% chance of losing $200 and 50% chance of losing nothing

Psychology studies have shown that we prefer the latter.

Similarly, in software development, suppose we were given a choice to:

• Pay down technical debt, but definitely miss a deadline (a sure loss)
• Deliver on-time, but risk a serious miss in the future (chance of losing nothing)

We also prefer the latter.

Concreteness bias

Concreteness bias refers to the human tendency to prefer the concrete and visible over the abstract and invisible. For example:

• It is more painful to lose $100 in your wallet than $100 in the stock market, even though it is the same amount of money
• Radiologists perform better at detecting tumors when given a picture of the patient along with an X-ray, than when given an X-ray alone

Similarly in software development:

• New features and defects are concrete and visible
• While technical debt consists mostly of invisible issues inherent in software development

Hence, we unconsciously prefer new features and defects, even when we believe technical debt could be more important.

Technical debt in the field

So there’s good reason for developers and architects to be concerned about technical debt. Several research studies back that up.

First, lack of awareness of technical debt is a leading concern for developers. Second, architectural debt is the prevailing concern for technical debt.

In one study, researchers at Carnegie Mellon surveyed 1,831 software engineers and architects at 3 large organizations⁷. They found:

• 79% believed that lack of awareness of technical debt is a problem
• 65% reported no defined technical debt management practice
• 42% said executives and business managers were largely unaware of technical debt.

Architecture debt is the prevailing concern

In addition, in that study, practitioners viewed bad architecture as:

• The leading source of technical debt; and
• Twice the concern of the next three: overly complex code; lack of code documentation; and inadequate testing

In another study, researchers interviewed 54 Finnish programmers⁸. They found:

• Architecture and documentation are the leading sources of technical debt
• Coding standards are most important to addressing technical debt, followed by simple design, refactoring, continuous integration, and test-driven development

Lastly, it is important to take project delivery methodology into account. one study examined the roots of technical debt in both agile and waterfall projects⁹. They found:

• Increase architectural debt by focusing on features over the entire product; and
• Decrease architectural debt by offering an iterative process to track and take care of the debt

A path toward better managing technical debt

So what can we do? Here are three ideas to better manage technical debt:

1. Adopt a team mentality;
2. Ensure both awareness and alignment; and
3. Adopt a test analytics platform

Adopt a team mentality

First, managing technical debt is the responsibility of the entire team. Collaboration between development, testing, and integration improves results.

For example, consider test automation. There are five ways to make test automation effective at reducing technical debt¹⁰

1. Collaboration between dev, test, and CI is essential for the success of agile test automation.
2. Automate tests for each layer of software, than striving for 100% of a single layer.
3. Keep test automation simple and reusable for the project team.
4. Automate stress and security tests early to reduce risks and rework.
5. Use test automation for documentation and information feedback. Tools like Jenkins, Hudson, and Testlink can help.

Furthermore, code reviews are a popular techniques to avoid technical debt. One study on code reviews in the open source community showed¹¹:

• Feedback offered 75% on maintainability and 25% on functionality
• Number of changes triggered by a review was independent of the reviewer’s experience level (so junior developers take note!)

Another study on continuous delivery at ING Nederland investigated refactoring as a result of feedback¹². The researchers showed:

• Refactoring was performed 16% almost always, 30% often, and 37% sometimes
• During code reviews, finding bad smells and sharing code knowledge was more prominent than finding defects

Ensure both awareness and alignment

Next, distributed agile teams need both awareness and alignment to manage technical debt. Consider the following 2×2 matrix to map organizational awareness and alignment¹³:

In this framework, distributed teams are aware when they:

• Have a common understanding of the meaning of technical debt and are mindful of it; and
• Recognize technical debt either instantaneously or during reviews and retrospectives

Teams that are unaware neither have a common understanding about the meaning of technical debt nor are they mindful of it.

Distributed teams are aligned when they:

• Have a common understanding about how to organize and manage technical debt; and
• Pay it off to increase business value delivered to stakeholders

Distributed teams aren’t aligned when they:

• Lack a common understanding on how to organize and manage technical debt; and
• Become reactive in paying off technical debt

Adopt a test analytics platform

Finally, a test analytics platform can reinforce a whole-team mentality. Test analytics combines continuous improvement, feedback mechanisms and automated testing.

For example, an early-stage startup used test analytics to overcome technical debt by adopting a whole-team mentality to unite continuous improvement, feedback mechanisms, and automated testing¹⁴.

In the 25-person company, distinct dev, QA and Ops teams resulted in extra handoffs. There was no crisp definition of done, and developers considered automated testing to be an uninteresting chore.

To curb mounting technical debt, the company adopted behavior driven development through a series of training workshops to formalize the definition of done. Next, they used gamification to encourage writing automated test scripts. Finally, they created a test analytics platform consisting of an analytics dashboard, gamification UI, and test probes (to capture test results).

As a result, the company significantly improved software quality and even launched the test analytics platform as an open source product.

Conclusions

Since my initial collision with the painful realities of managing technical debt, I’ve become much more proactive on digital programs. Recently, I was advising the Senior Director of Test Strategy at a large healthcare company on an upcoming digital transformation initiative.

He shared with me his ambitions in crafting a new image of testing as the tip of the spear, rather than the tail of the dog. However, he encountered roadblocks within the silos of the IT department.

Therefore, we created a set of testing dashboards to monitor technical debt. Over time, we showed that better managing technical debt caught defects more quickly and led to fewer defects overall. Since then, testing is now an equal leader on all subsequent digital programs at the company.

Managing technical debt is not a luxury. It is the most consistent way to stay one step ahead.

1. W. Cunningham, “The WyCash Portfolio Management System,” Proc. OOPSLA, ACM, 1992.
2. M. Fowler, “Technical Debt,” blog, 2009; http://martinfowler.com/bliki/TechnicalDebt.html
3. P. Kruchten, R. L. Nord and I. Ozkaya, “Technical Debt: From Metaphor to Theory and Practice,” in IEEE Software, vol. 29, no. 6, pp. 18-21, Nov.-Dec. 2012.
4. Gene Kim, Patrick Debois, John Willis, and Jez Humble. 2016. The Devops Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press.
5. “World Economic Outlook Database, October 2020”. IMF.org. International Monetary Fund. 13 October 2020.
6. A. Martini and J. Bosch, “The Danger of Architectural Technical Debt: Contagious Debt and Vicious Circles,” 2015 12th Working IEEE/IFIP Conference on Software Architecture, Montreal, QC, 2015, pp. 1-10.
7. N. A. Ernst, S. Bellomo, I. Ozkaya, R. L. Nord, and I. Gorton. 2015. “Measure it? Manage it? Ignore it? software practitioners and technical debt.” In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). ACM, New York, NY, USA, 50-60.
8. J. Holvitie, V. Leppänen and S. Hyrynsalmi, “Technical Debt and the Effect of Agile Software Development Practices on It – An Industry Practitioner Survey,” 2014 Sixth International Workshop on Managing Technical Debt, Victoria, BC, 2014, pp. 35-42.
9. A. Martini, J. Bosch and M. Chaudron, “Architecture Technical Debt: Understanding Causes and a Qualitative Model,” 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications, Verona, 2014, pp. 85-92.
10. E. Collins, A. Dias-Neto and V. F. d. Lucena Jr., “Strategies for Agile Software Testing Automation: An Industrial Experience,” 2012 IEEE 36th Annual Computer Software and Applications Conference Workshops, Izmir, 2012, pp. 440-445.
11. M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens. 2014. “Modern code reviews in open-source projects: which problems do they fix?” In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR 2014). ACM, New York, NY, USA, 202-211.
12. C. Vassallo et al., “Continuous Delivery Practices in a Large Financial Organization,” 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), Raleigh, NC, 2016, pp. 519-528.
13. R. Bavani, “Distributed Agile, Agile Testing, and Technical Debt,” in IEEE Software, vol. 29, no. 6, pp. 28-33, Nov.-Dec. 2012.
14. O. Liechti, J. Pasquier and R. Reis, “Supporting Agile Teams with a Test Analytics Platform: A Case Study,” 2017 IEEE/ACM 12th International Workshop on Automation of Software Testing (AST), Buenos Aires, 2017, pp. 9-15.