
Ask HN: As a CTO, what is your most frustrating problem with technical debt? - waterlink
_or as a person in a CTO-like role_<p>Ah, the technical debt.<p>While the concept is sweet from the business perspective, the result of leveraging technical debt, most often, is a big mess.<p>The technical debt is accruing every week, after each iteration, sprint, what have you. Your developers are whining about it, and they want to do a significant rewrite or solve it with microservices altogether.<p>And you know this rewrite, like others in the past, will be a significant sink of time, and likely a catastrophe.<p>This is how I see it from my humble developer&#x2F;team lead perspective. I’m really curious to know how people in CTO roles perceive it.<p>I agree that this is a useful quality lever that can get the whole company out of a bind, or could let business leverage a market opportunity before the competitors do.<p>What I’ve found is that people in various project-manager-like roles tend to overuse this lever. I believe this lever should be used only when there is a significant gain for the business, and the debt should be paid shortly after that.<p>What I’ve personally seen, and what I’ve been told by my fellow developers, is that it is common to use the technical debt lever just to:<p>- reach the “sprint commitment,”<p>- meet the artificial deadline,<p>- increase “developer’s productivity,”<p>- show manager’s power.<p>These don’t sound like critical high-value goals to reach, given the price of the Technical Debt.<p>That is my perspective. Now, what do you have to say?<p>As a CTO (or in a CTO-like role), what was your most consistent pain, frustration or problem with Technical Debt in your company in the last 2 years?<p>Have you ever had any?<p>Thank you for your time reading this and giving a thoughtful response. :)
======
ghiculescu
I think a lot of the time when a developer shouts “technical debt” what they
are really shouting is “code someone else wrote that I’d rather rewrite than
understand”. (The rest of the time is the same but they’ve understood it
enough to think it’s a disaster area.)

I have found it’s best to not take tech debt complaints very seriously and
instead look at actual success metrics. For example if every change to a bit
of code introduces new bugs then that might be a reason to tidy it up.

~~~
mabbo
> “code someone else wrote that I’d rather rewrite than understand”

I'm not disagreeing, but there is something to be said for writing code that
any idiot can understand so that future developers don't even _want_ to
rewrite it, and can maintain it.

I've landed in code bases on both ends of the spectrum, and I never asked
management to rewrite the well-written ones.

~~~
omeid2
In a wicked way, people who write that kind of code never get credit for it,
and often lose leverage of their job.

I am not saying that writing convoluted code is good, no, but I can tell, even
if based on personal experience, that barely anyone talks about "wow, this is
such a straight forward code" or gets promoted for "everyone can understand
your code!".

It is a weird dynamic that gets worst by having non-technical managers, and
the reason why I believe team leaders should be hands-on in software
development and involved with code review.

~~~
waterlink
Let me get this straight:

Person A, a programmer, who writes easy-to-understand code, but takes a bit
more time to finish their tasks

Person B, a programmer, who writes dirty code, but takes much less time to
finish their tasks

The current non-technical manager will see only one part of the picture and
promote Person B. Moreover, Person A might get in trouble, and pressure to
“compromise on quality.”

Within a short-term, it is understandable.

Within a long-term, this is a disaster scenario!

So should the “code quality” be part of a performance portfolio of the
developer when it gets to promotion time? And how do we get there?

~~~
monocasa
I've also seen

Programmer A writes straightforward code that's easy to change and understand.

Programmer B writes convoluted "clever" code that completely falls apart when
requirements change, or even just in production because their clever solution
didn't account for all of the edge cases.

Now Programmer B has to save the day consistently, and therefore is seen as
the person saving the company, when the problems were all of Programmer B's
construction to begin with.

~~~
linuxftw
I would expand upon this and say that Progammer B doesn't ship the code at
all, it wouldn't pass code review muster.

Unless we're talking about a seriously silo'ed org, or Programmer B is
actually "Team B"

~~~
monocasa
A lot of teams are a cult of personality around one developer. Particularly if
there are a bunch of juniors on the rest of the team, Programmer B can usually
get anything through code review with the 'ol "of course it's complicated,
that's why _I_ am the one working on it. It's not for mortals to understand".

~~~
hinkley
Treat every question about your code as feedback. If multiple people ask the
same question you have a problem that needs to be fixed. Documenting the
gotcha should be your last resort. Implying incompetence isn’t even in the
board.

------
moxious
I've been the CTO at two tech startups. Here's how I think about it.

"Debt" is a good metaphor because by accumulating it, you get advantages right
now in speed and ability to focus on other things, in exchange for having to
pay it off later. You can either pay it off by stopping what you're doing
(uncommon) and fixing stuff, or you can do lifetime installment payments of
more complexity, bugs, etc.

Tech debt is just one aspect a CTO has to manage. Time to market is pretty
damn important. Features are important. Many other things too. This is not to
say that tech debt should be ignored, but elegance of the system is not the
paramount concern. Just as startups borrow money and take financing to move
quickly, so too a responsible amount of tech debt can be a good thing.

In engineering we tend to over-emphasize how important the technical qualities
of the system are because that's what we see every day. There are a great many
things that are also important, and various business scenarios where they
become MORE important than tech debt. But if you're in the stage of trying to
find product-market fit, you probably have bigger fish to fry.

The biggest tech debt I've struggled with always comes from the same patterned
source. You think a system is intended to have a certain feature set for a
certain user base, and so you make a set of architectural decisions to match.
The world moves on quickly, you adapt, pivot, whatever you want to call it,
and suddenly your carefully made architectural decisions are no longer correct
or wise for your new scenario, but there is no time to go back and have a "do-
over".

So in essence, failure to be able to tell the future, paired with a constant
need to move forward, is the source of the problem. Both of those things are
going to keep happening to startups forever.

~~~
stingraycharles
In other words, it’s not a problem to be avoided, but rather a problem to be
embraced.

If this is indeed true, how does one fully embrace it? Optimize for easy
refactoring? What does this look like in practice?

~~~
moxious
No, I wouldn't put it that way. It's not a problem to be embraced, it's just
another variable to manage.

Just as there is the trilemma of "better, faster, cheaper: pick 2" there are
others as well. Speed and tech debt are balanced against one another. It's not
that I think we should embrace tech debt. I actually rather hate it. But
sometimes I'm willing to trade to get to a business objective that is not
related to technical elegance.

In terms of how to pay it off, it may sound like a cop-out but it just
depends. The core problem is that things are always changing and we can't
predict the future. So staying supple and flexible I think is the way to go.
Pay off as much of it as you can when you can, but I think the main point for
me is to keep the eyes on some business objective (where tech debt appears as
only one variable in the equation) -- the main point is __not __to keep your
eyes on the tech debt all the time.

------
notender
If you believe technical debt is developers wanting the code to be "perfect",
you completely misunderstand what technical debt is.

Technical debt is a collection of choices you and your team make that can have
real world impacts on your companies ability to deliver features in the
future, can open your company up to hacking, and could make your software less
competitive than the rest of the market.

Technical debt is not a loan, it is an anchor. Instead of deferring the debt
payment until later, you pay for it with every new feature and eventually your
development team will slow to a crawl or the product your building will no
longer be cost effective due to rising maintenance costs.

Technical debt is a unknown risk. Look up any "X company was hacked" story and
they have all come down to bad choices made by the product team. It could be a
developer on your team left there machine unsecured or put in some bad code,
or it could be a deferred decision to fix that technical debt until later.

Technical debt confines the space in which you can operate. If the market
pivots, but your software is too rigid to keep up with the changing shape of
the market you will lose market share or the ability to capitalize on new
things.

All companies will have to make hard choices when it comes to technical debt,
and no product can be debt-free. But understanding that technical debt is more
than your developers whining about code is paramount to being an effective
leader in your organization.

~~~
Zimahl
> Technical debt is not a loan, it is an anchor.

This is a great analogy. You can still sail a ship with one anchor, maybe two,
but eventually you are going to have so many anchors that ship isn't going to
ever get to its destination.

------
sudhirj
I try to use an economic model for all software. All code is a loan against
taken against entropy (a liability).

Best solution is no code or a simplification of the problem to existing
solutions. This results in zero liability, while the revenue the solution
generates is pure assets.

When a code liability does need to be incurred, consider both the amount of
code written (the principal of the loan) and the architecture of the solution
(the rate of compound interest). A good programmer will work to reduce the
principal, an excellent one will work to reduce the rate of compound interest
with a simpler architecture.

The architecture that is does more by adding complexity and conditionals /
code contains a horribly high rate of compound interest, even if it seems
easier to do at first.

Architectures which do more / everything by virtue of being simple may be more
difficult to implement and understand (especially without some background
knowledge) but can have a vanishingly small rate of interest. Double entry
bookkeeping, graph theory, pipelines, normalisation/denormalization,
map/reduce, unidirectional data flow etc.

For any serious business investment (not an experiment or proof of concept),
architecture as rate of interest always trumps quantity of code as principal.

Order of priority is 1\. Don’t take loans 2\. If you must, design until you
have a low and manageable rate of interest. Revisit often. 3\. Reduce amount
of code.

~~~
hire_charts
> Best solution is no code or a simplification of the problem to existing
> solutions. This results in zero liability, while the revenue the solution
> generates is pure assets.

Not exactly _zero_ liability, but the source of the liability is pushed
outside the bounds of your own code, such that the burden is shared among far
more people than you can employ on your own. Point being, you may still find
yourself contributing code to these shared solutions, but that's a good thing,
because such external work will have a wider-reaching impact than anything you
build internally.

------
snowwrestler
I run a bunch of websites on top of popular, open-source CMS software like
Wordpress or Drupal.

An under-appreciated source of technical debt in those sites are plugin/module
upgrades--the ones that are not security patches. Most developers and agencies
seem to take the approach "if it's working for us, and not a security issue,
leave it alone."

That approach works fine until there _is_ a security patch. Then you have to
apply it fast, and guess what--it comes with all the previous code changes you
chose to avoid. Now you're getting all those changes all at once, and under
time pressure.

Keep your plugins and modules up to date! It might mean some level of ongoing
development to deal with changes in functionality. That's a good thing. That's
a muscle you need to flex in order to keep it in shape. If you only update
modules when there's a scary security issue, some of that technical debt is
going to come due when you least want it to.

Another underappreciated source of technical debt is the differing preferences
among developers and agencies on how to use the CMS + plugins/modules to build
a site.

Ok you're building a Drupal site... do you build your layouts with custom
template files? Context module? Panels and Panelizer? Paragraphs module? Do
you handle embedded media in fields or WYSIWYG? Do build your sidebar with
blocks or with Views + nodes + Display Suite?

You can build the same website different ways, and it will work fine. But a
legitimate choice can often look like awful technical debt to someone else,
who would have made different choices. You have to take this into
consideration when hiring new developers or agencies to work on an existing
site. Otherwise you can waste a lot of resources rebuilding functionality with
no discernible improvement to the end user.

~~~
amelius
That's what backports are for.

[https://en.wikipedia.org/wiki/Backporting](https://en.wikipedia.org/wiki/Backporting)

~~~
snowwrestler
Backports are very rarely provided by the community for older plugin/module
versions.

Backporting by hand, on your site, makes technical debt worse. Now, not only
are you not up to date with the community code, but you've got extra custom
code that will have to be integrated or torn out to catch up.

In terms of competencies: if a team is struggling to integrate and deploy
community-provided code, how good a job are they likely to do writing,
testing, and deploying custom code? Especially in a hurry.

That's what I mean by flexing the muscle. A team that updates their site's
code frequently is going to be a team that understands their application
better, has more complete test coverage, and deploys with more confidence. And
if custom code is required (like a backport), they're going to be better
equipped for that too.

------
winslett
There are a few side-affects of technical debt that drive me nuts. These come
from companies where developers are autonomous, self-choosing of time
investment:

* continuous complaint without action (i.e. pretending to be a victim of the system you’ve built)

* lack of measurements for a system (technical debt or not, how is it working? Are you improving your measurement system after each failure to further understand the system?)

* unbuilt is better than unmaintained (this builds off the previous two — tell me what we learned from our previous attempts. If you didn’t understand the quality of the previous attempt, then how do we know if we actually improved?)

I’m down with fixing technical debt, just show me some metrics that prove that
it was an improvement.

~~~
waterlink
> pretending to be a victim of the system you’ve built

Nicely put! I haven’t thought of it this way.

> Are you improving your measurement system after each failure to further
> understand the system?

Oh my god. Yes. This is a quote that made my day.

So far, it feels like we need to:

1\. Have a concept of area (and maybe sub-area) of the code-base

2\. Do frequent checks how painful it is to work with a certain area. Maybe,
after every feature (ticket, what have you), make developers record how they
felt about the areas that they’ve touched, and why. For example, this was
really painful because this class is just too coupled to this other one or
something like that. Track this information.

3\. Track frequency and size of changes required to different areas in the
code right now (+ anticipation in the next period, whatever it is).

4\. Regularly have a technical retro, where you look at the areas, and
prioritize areas that:

    
    
      * give the most pain AND
      * need to be frequently changed now and in the future
    

5\. Then go through the list of problems and choose the most horrible ones to
resolve

6\. Schedule time to resolve these, just like we do with features/tickets/what
have you.

This is just a draft of what comes to my mind.

~~~
invertednz
This is very similar to what my startup is looking to help with.

------
1_800_UNICORN
For the last 5 years I've worked primarily with startup CTOs and enterprise
directors/VPs to build teams practices XP, lean product development, and user-
centered design.

My advice to them, and my philosophy on this topic, is that technical debt is
just that- debt. It's not a dirty four-letter word to be avoided, it's
something to be managed the way you manage actual financial debt. There are
some projects where the software you build will be self-contained (no
dependencies), will require minimal changes in the future, or it may have a
short shelf life (e.g. it's a stopgap before you cut over to a new system). In
these scenarios, accruing technical debt isn't the worst thing in the world if
it allows you to reap business value faster.

When you're building software that's going to live long-term, or that drives
core parts of your business (and therefore will need to change frequently as
your business changes), technical debt needs to be tackled early and often.
It's ok to accrue technical debt in the short term ONLY if you make that
decision consciously knowing that you'll need to address it later (e.g. I had
a client that needed to release their software for a trade show, so we accrued
tech debt and created tasks to track every piece of debt we'd want to tackle
later, and then prioritized that work immediately after the trade show).

That said, it's also important to be clear about what constitutes tech debt vs
what is over-optimization. Outside of libraries and frameworks, I don't really
need the software to be "perfect". I need it to be clear enough, tested
thoroughly enough, and easy enough to change to maintain a high velocity. If a
part of the code has become spaghetti-like, or if we see code duplicated more
than 3 times, or if we have disparate code that's too closely coupled to
easily make changes, that's technical debt worth tackling.

The vast majority of organizations I've worked with typically think much
shorter term, which is why they find themselves repeatedly accruing technical
debt to the point where changes to the system occur at a snail's pace. It
takes a lot of discipline, and coordination between IT and business, to
practice what I'm describing, but that's the ideal that I strive for and
advise my clients to strive for.

~~~
lkrubner
" _It 's not a dirty four-letter word to be avoided, it's something to be
managed the way you manage actual financial debt._"

As others have pointed out, technical debt operates the same way an unhedged
option works. It can go catastrophically debt. Compare that to issuing bonds.
If you issue $10 million in bonds, you know exactly how much you will have to
repay, and you know the interest, and you know when you will have to repay it.
With technical debt, you don't know when you'll have to repay it, and you
don't know how much it'll cost you. You just go along knowing that someday
something catastrophic might happen. See this previous discussion:

[https://news.ycombinator.com/item?id=8777237](https://news.ycombinator.com/item?id=8777237)

~~~
waterlink
Technical debt is very similar to financial debt.

Don’t get fooled. There is very small, and very important difference:

With financial debt, the business takes on it in hopes of delivering more
value quicker that will pay for interest and principal, and have a lot of
(monetary) value left after.

Financial debt, if anything, is speeding the process of delivery of value up.

On the other hand, technical debt is slowing the delivery down. And this
delivery is what business hopes to speed up.

Well, maybe it makes this one iteration/release quicker, but it slows down the
next one, and next one after that, and so on.

So, it seems that they are deceivingly similar, but you need to adapt the
tactics and strategies.

~~~
smu
I fail to see the difference. Financial debt is also a brake on future
activities, in the sense that you have to pay back with interest, so less
funds are available for other activities.

So in your words: it makes this one iteration quicker, but slows down the next
one...

~~~
waterlink
I see what you mean, and I agree partially.

Follow my thoughts (and please do poke holes in them!):

Financial debt allows you to deliver more -> sell more -> pay down debt
quicker -> deliver more (if you got it right and haven’t failed).

So the money itself becomes a mechanism by which you deliver faster, and, as a
result of faster delivery, earn more money to deliver even more and faster.

Now, the technical debt.

Earning more money from more feature delivered in current iteration might
result in more revenue NOW. But does it result in more/faster delivery after
that revenue happened?

Not really.

You can potentially hire more people. But they will only slow the team down at
first. There is a delay between “more people” => “faster delivery.”

So financial debt can enable a positive reinforcing loop. Can technical debt
do that?

------
cntlzw
The most frustrating part is explaining technical debt. From a business
perspective you have a running software. You can sell it - it has all the
features needed. Now you come along and tell me you need to fix something,
that no client can see, that does not add any features and it might even lead
to bugs. And you need two people and x weeks. I can imagine how "technical
debt" is perceived as "snake oil" or "developer fun time" for management.

To avoid frustration it is best to fix technical debt in small steps, mix it
in with others open issues. Devoting developer time for a big technical debt
rewrite is hard to communicate.

~~~
kylestlb
The way that I've explained it with higher degrees of success is: "Feature X
is used by a high percentage of users and also contributes to a high amount of
bugs. We can invest two people and x weeks to bring those bug numbers down and
product quality up."

~~~
cntlzw
Totally agree with you. As other people commented here. The key to
communicating about technical debt is numbers. You need some metric to at
least show that you have a problem and a number to show that you improved
quality after reducing technical debt.

------
kashyapc
Speaking of technical debt, you might want to remember the following 'handy
rule' from the book Team Geek[1], chapter "Offensive" Versus "Defensive" Work:

 _[...] After this bad experience, Ben began to categorize all work as either
“offensive” or “defensive.” Offensive work is typically effort toward new
user-visible features—shiny things that are easy to show outsiders and get
them excited about, or things that noticeably advance the sexiness of a
product (e.g., improved UI, speed, or interoperability). Defensive work is
effort aimed at the long-term health of a product (e.g., code refactoring,
feature rewrites, schema changes, data migra- tion, or improved emergency
monitoring). Defensive activities make the product more maintainable, stable,
and reliable. And yet, despite the fact that they’re absolutely critical, you
get no political credit for doing them. If you spend all your time on them,
people perceive your product as holding still. And to make wordplay on an old
maxim: “Perception is nine-tenths of the law.”

We now have a handy rule we live by: a team should never spend more than one-
third to one-half of its time and energy on defensive work, no matter how much
technical debt there is. Any more time spent is a recipe for political
suicide._

\- - -

Previously discussed here:
[https://news.ycombinator.com/item?id=16810092](https://news.ycombinator.com/item?id=16810092)
("A Taxonomy of Technical Debt")

------
swalsh
I'm one step below the CTO, but here's my perspective for what it is worth. It
really depends on the stage of development your company is in. When you're
first starting out, writing "good quality code" is about the worst thing you
can invest in. The fact is, the code you're writing is almost certainly the
wrong thing. A new startup is a hypothesis on a business model, and the first
bit of code is to make the most basic tests of the hypothesis. It's VERY rare
you get it right your first try. Writing good quality code is a long term
investment, but if there's no one willing to buy it, it's a bad investment
that will never have an ROI. On the other hand, if you're rewriting the system
because now you know the market, and you know what it wants and you're scaling
to meet it, now tech debt is bad. I think it's critical to understand the
first system can and probably will be thrown away, so it makes little sense to
build it to "scale".

~~~
jestar_jokin
Given the uncertain business direction, you will definitely need to make the
code easy to change. Low quality code inhibits that.

------
revel
Since there are already quite a lot of responses to your original question
I'll instead address a specific aspect of your post: when you have an old
legacy system that you don't want to build on any more it is almost _never_
the right decision to try a one-shot migration or rewrite.

I have seen so many people try and solve technical debt with a full rewrite
and it just never works. The better way of solving this problem is to
gradually peel off individual use cases and API calls. It doesn't matter if
your legacy system is riddled with bugs and is impossible to build on; it's at
least a known quantity. Your new system will likely have less bugs in it, but
until it's been used in the wild for a while you won't know where those bugs
are. If your new system has a systematic problem, you are in some serious
trouble, but if you migrated over slowly at least it will be manageable.

------
watwut
People here talk about technical debt as equivalent to financial debt, but I
think it is wrong. Differences:

1.) Financial debt does not slow you down, technical debt does slow down work.
Depending on how messy the system is, it can kick as soon as 3 months into
development. It can make project fail or slow to develop and it is insidious -
it is sometimes hard to see that things could be faster from CTO position.

2.) If you have money it is easy to pay financial debt and it happens
instantly. Even if you have time, it may be hard to impossible to fix
technical debt. The engineers might just create bigger mess in their attempt
to clean things up, they may cause bugs, miss whole features etc. But most
often, they just don't end with all that much cleaner system, because the mess
is result of broken process or other systematic reasons or missing knowledge.

3.) It is easy to know when you have financial debt. It is harder to know (for
CTO) when you have technical debt. Some engineers complain about pretty much
everything someone else wrote. Many decisions are actually subjective calls
where there are multiple valid options and the issue is not so much "it is
bad", but "I dont agree with decision". They will never all agree. Some
technical debt complains are attempts to make oneself look like more attentive
coder, to excuse perceived length of development or part of power struggle
between engineers.

~~~
collyw
> Financial debt does not slow you down, technical debt does slow down work.

It slows down your spending, or it ought to.

Instead of spending money, technical debt is spending time.

~~~
waterlink
Interesting.

Then, in one month of time, you can potentially have 5000$ or 5M$ in finances
available.

In time though, in one month, you can have only one month.

It is true, it is spending your time. But it seems like this time is much more
precious resource than money.

What do you think?

------
Zyst
The Effective Engineer[0] has a chapter on technical debt, where he goes into
how a lot of crappy code decisions are there for a reason, and how rewrites
can be chaotic. If you have the time, I'd recommend giving it a read.

I frankly got a lot more pragmatic about asking for rewrites after reading it,
and I feel it helped me to grow (mature?) as a developer.

Either way, I think there's a need to balance the need for employee self
actualization needs which they often push as rewrite requests "Oh since we're
rewriting this, we should (do it in)/use/etc X instead". I have often realized
that a lot of requests to rewrite something are really tinkering desires
camouflaged as a business related request, which is not to say that the code
that does exist could have problems, it could, but having a period of debt
repayment would improve it as well. So finding a way to allow your employees
to tinker without letting their desires torpedo your products would be
positive.

Either way, it's a complex subject, and I don't really think there's a single
"right" response to it. Best of luck!

0: [https://www.amazon.com/Effective-Engineer-Engineering-
Dispro...](https://www.amazon.com/Effective-Engineer-Engineering-
Disproportionate-Meaningful/dp/0996128107)

EDIT: I'm not a CTO, I'm a developer.

------
cdevs
I notice at my job there were a lot of i need feature x to which we commonly
replied i would like to build x but the horrible old y system is stopping me
or we would just make things much worse to try and meld feature x into
legacy/messy y system. I was a dev dealing with the legacy systems at my
company for years and then i took over management and said we need to remove
or clean up these legacy weak points so we can build features faster and more
maintainable, so yes we started at the first day of this year cleaning up from
the worst / weakest links first on. In 6 months we are done using 2 new hires
to complete new work while 2 senior devs did clean up. 6 months compared to
years of spinning tires was well worth it. Eventually the CEO gets sick of
hearing excuses of why we cant build this or that or are wasting 50% of our
time fixing bugs. Productivity is soaring now for multiple reasons at my
company, testing, documentation, removing old systems and even though we have
a small team every senior dev is leading a junior dev. Side note this is the
first time we have hired junior developers at this company and its been a big
pay off giving a lot more free time to the senior guys to work on the most
important things.

------
alexmorse
I don't have much frustration with tech debt itself. It's a normal,
acknowledged, often understood and predictable side effect of moving fast OR
weighing tradeoffs.

I actually get pretty annoyed when it's conflated with product debt. I don't
know if that has a real definition, but for me that's where your
market/audience has grown to need things you did not envision/never built.
This is often labeled as tech debt, even though it has nothing to do with your
coders. This is an organizational issue, and one I haven't fully conquered
yet.

Tech debt should not be accruing every week, it should be a conscious
decision. MVP's are riddled with tech debt, intentionally. You want to get the
thing out there and start gathering feedback/information, you acknowledge you
don't know the right thing to build, and will do so once you have more info.

I'm also annoyed at the junior dev that doesn't take time to understand a
system, and shouts that it needs to be re-written. Most times that's just
because the dev is junior, or lazy. It's actual tech debt when the system is
too convoluted to understand, or you haven't documented.

I have no idea what "sprint commitment" or "manager's power" are supposed to
mean. They both sounds kinda sick.

If you know a rewrite is going to be a time sink and catastrophe, why on earth
would you do it? Also sounds sick.

Your org should be focused on business results. If you're focused on optics,
things are not healthy.

You are not in a race with your competitors unless you're in a market that is
itself racing to a commodity. This is a business strategy problem vs an
engineering one.

Technical debt seems far less a problem to me if you have a moderately capable
software engineering team. It happens, yeah. But as your experience grows you
know the proper balance of paying it down vs new work. It comes down to
measuring quality of your services from an end user perspective vs how much
work on engineering and support teams to maintain the desired quality.

~7-8 years cto'in

~~~
ben509
> I'm also annoyed at the junior dev that doesn't take time to understand a
> system, and shouts that it needs to be re-written.

That's something that your senior devs should be correcting through training
and mentoring. Junior means he doesn't know any better, after all.

Speaking as a senior dev, one great method to impart this wisdom is to let the
junior dev try and do it and see for himself what happens.

------
sonaltr
One big issue I have (and I'm trying various solutions around it) is code
upgrade.

Let's say we are doing something one way - step 0, and we realize "hey, guess
what' there's a better way" \- let's call it step 1). Now going forward all
new code is step 1, and touches to old code is step 0. Later you realize -
there's an even better way - let's call this step 2. Now going forward you
have step 2, step 1 and step 0. This process continues - but people who worked
on this leave - leading to technical debt. This carries on for a few
iterations and now you are on step 50 (49, 48, ...).

I have no idea how to deal with this problem.

I personally feel more micro-services would be better - but I also don't want
to jump the gun since we were not too big a company at that point (and being
profitable with our runaway took all priority)

~~~
bradstewart
It's a hard problem to deal with. I try to make sure the "better" way is
"better _enough_ " to warrant a change, knowing the technical debt that will
ensue.

------
snarf21
The biggest problem is that Product and/or Sales are looking for constant
growth via constant new features. It is impossible to get them to truly
understand that by meeting some arbitrary deadline that you've had to take
shortcuts. To their mind, done is done and now we need the next Y so they get
their Q3 bonus.

The next problem is that too often the tech team wants to do a complete
rewrite every 12 months because they want to use hindsight to remake last
year's system/feature today. The issue is that in another 12 months this
rewrite suffers from the same problem and now has several more things to
consider. Some technical debt will always exist and I would argue always
necessary. Perfect is the enemy of good enough and software that isn't live
isn't adding value to the customers or company.

My perspective for this was to treat some percentage of work each quarter to
resolving technical debt but do this in an agile prioritized way. Just like
always focusing on the feature that will add the most value to the next
release, also pick the biggest manageable issue that will reduce the debt the
most to the next release. In this way, you know certain technical debt will
never disappear but you have a ranked list to always decide the things that
are most urgent and correct them before the house falls down. If your team is
large enough, it is also helpful to have a person or standing task to update
and document this list with WAG options and costs. Of course, as in all
things, ymmv.

------
3pt14159
Technical debt should be quarantined, not structural. That's the one, most
important thing to take away. A really messy view with tangled if statements
and hacks to get things to work isn't great, but it's leagues better than a
really messy controller because it isn't going to introduce ACL errors. A
really messy controller is leagues better than a fucked schema / data model
because it doesn't spread the poison across every endpoint in your
application.

Some easy wins are to default to making something a "many", even if it is only
a "one" or a "one or none" because it makes adding a second case much easier.
For example, a user should have multiple roles at multiple projects by
default, so business users can do fine grain control over permissions.

Another useful trick is to have default behaviour, but introduce a set of
tables for custom settings. This makes it easier to make changes because you
don't need to run a migration to add a column to a table, you just write a new
record to the "user_overrides" table.

But at the very end of the day, taking on technical debt of any kind can be
warranted. It all depends on the stakes. If you need to get something done in
a week to close a company-saving deal, then fucking do it. Fix it later. The
implied interest rate might be 1000%, but if it saves the company it is worth
it.

~~~
katnegermis
One of my friends came up with a tongue-in-cheek name for your "default to
many"-advice: [https://theharemrule.com](https://theharemrule.com)

After being given the advice couple years ago, I found it to be applicable to
quite a number of scenarios me and my colleagues faced. In my experience its
pretty solid!

~~~
3pt14159
Yeah, it's one of those tricks that I find a lot of developers in their 20s
haven't learned.

There is another trick that I use when I don't control the dev team. For
example, say the company has outsourced a pricing app to a third party dev
shop, in this case any decision I can make that will make the app more
"table-y" I make. So instead of defining control flow to determine price for
any given object, what I do is I get them to make a "rules engine" that can
operate on the major variables that are typically used to make decisions. That
way fine tuning changes can be done on our side, and it doesn't rack up
billables. I think this is called "table based design" but Googling for it
isn't showing anything.

------
kirubakaran
I've been thinking how technical debt can be flagged / measured, and then
managed. That is, a tool that can do a combination of:

Manual: Perhaps the person doing the code review can negotiate with the
developer and agree that the code needs to be cleaned up at some point soon,
but needs to be merged as is right now due to business needs, with some
sensible score for the magnitude of the debt. When the debt does get paid
later (stop laughing), the actual effort can be correlated with the debt
score.

Semi-Automatic: The tool can take diffs from a pull request and show it to
some other developers in the company and poll them on some subjective
attributes like clarity etc. The score will be normalized based on other
scores given by the person. Once in a while the app can even show nonsensical
code in this poll to see how that is scored.

Automatic: 1. Flag the files that change frequently with issues, other pull
requests, or 2. when there are long conversations and further commits to a
file before a pull request affecting some files are merged (while accounting
for some reviewers approving anything vs some reviewers nitpicking
everything)... I have several more ideas like that.

Would you use a tool that does this? Do you already have something like that?
Please feel free to email me your feedback, if you'd rather do that:
techdebt@kirubakaran.com

~~~
thewanisdown
We treat technical debt the same as any other issue. Put an issue on the
board, decide on timing, assign it to a release, etc. I think the problem for
some is that they don't realize they are creating technical debt.

------
jillesvangurp
Technical debt is inevitable. If you ignore it, it becomes worse. I've never
been on a project without technical debt.

Of course not every little problem is technical debt. There are plenty of nice
to haves, we could do X, etc. style improvements that aren't really that
critical. My advice for those: just apply the boy-scout rule and improve
things when you work on them. For the bigger stuff, somebody needs to step up
and take decisions. Ultimately for big chunks of work related to technical
debt, that's a CTO decision that involves taking into account the interests of
both tech and business. A CTO must have the power to do this and be wise
enough to use that power responsibly.

Stuff to watch out for as a CTO is when seemingly simple stories snowball into
a lot of firefighting, bugs, or deployment issues. That's usually a good sign
something is wrong. If that happens a lot, you are definitely experiencing
technical debt. Firefighting eats away development budget. If your team spends
half their sprint diagnosing weird issues instead of getting stuff done,
cleaning that stuff up is probably time well spent.

------
nsfyn55
"Tech Debt" and "Code Quality" are myths. I know that is an unpopular
perspective, but allow me to explain. To start with, for me everything is just
a problem described in terms of its properties.

What are some examples of problems I have encountered that were worth solving?

1\. We have a server rendered site and an API. Changes to the server rendered
site require matching changes in the API code effectively duplicating our
efforts. If we re-implement the site as a single page app consuming the API we
can kill two birds with one stone.

2\. The lacking test coverage of a critical section of code matched against
the frequency of changes to that code results in a frequent regressions and
delays.

3\. Unifying two divergent implementations will allow us realize a shared,
multi-tenant deployment model for our partners. The load characteristics will
give us the same point of presence for half the hardware.

Why are these worth solving? Because they have objective rationales. The third
embodies a strategic direction with real bottom line consequences. The case
can be made "If you want this outcome here are the steps" Then the only
question becomes "Do we want this outcome?"

Many times "tech debt" and "code quality" are tossed around as easy
justifications. They lack the intellectual rigor or organizational context to
make a compelling case that its worth the opportunity cost. The person using
those terms generally has strong feelings that its the right path, but can't
really articulate why. In many cases there is no "why" and its more a of a
"just because"

In my experience "Tech Debt" projects that are actually undertaken or not
called "Tech Debt" they are called "Lets do this because its important to the
business and here is why" projects.

------
daxorid
The primary source of our technical debt is launch impatience.

Specifically, that most stakeholders in a feature or project accept "mostly
works, with the exception of rare edge cases" as sufficient cause for launch.
This usually takes the form of "get it out the door as-is so we can promote
it, and then refactor later".

This is actually somewhat reasonable in terms of balancing business and
technical interests, and I can see the case being made for doing it.

The problem, of course, is that "refactor later" never happens because there
is _always_ another project or feature demanding to be worked on. These same
stakeholders might allow a day or two after launch to refactor, but then
promptly decide that starting on Project B takes higher priority than
refactoring Project A (which, in their mind, "works").

The above scenario covers _at least_ 95% of our existing technical debt.

------
thewanisdown
The core trade-off is "what does the business need now" vs "what are the long
term goals". I only consider the technical debt that will block us in the
future.

At any given decision point, you need to understand when technical debt is
accrued and track it like any other issue. If you don't actively acknowledge
technical debt from the onset, you place your business goals at risk.

In general, I see that the most politically charged projects are also accruing
the most technical debt. They have high pressure, short time-frames, and scope
creep out the wazoo. Those take the most of my time, and are extremely prone
to failure unless there's a human shield to keep the heat off.

------
gesman
>> what is your most frustrating problem with technical debt?

Actually using the product that has a strong scent of technical debt behind
it.

Vendor provides updates (could call them minor or major) which are essentially
colored bandaid patches on top of most critical issues without any significant
improvements.

Likely caused by waves of temporary hit-and-run consultants hired to "save the
day" and each of them done that to minimize personal effort before collecting
the paycheck and leave.

Eventually the product needs to be completely rewritten or risk to become
obsolete due to healthy competition.

------
alexeiz
Poor or incomplete abstractions and code duplication. They usually go hand in
hand. What typically happens is that one developer needs to solve a problem.
He creates some abstraction (one or more classes) to help him solve the
problem. He tries to make his abstraction reusable, but it's hard to do if
there is only one use case for it. So the abstraction ends up with some
essential functionality missing. Or worse, the abstraction may be poorly
designed, so it's not reusable at all.

The second developer comes along with his own problem to solve. He looks
around and finds that abstraction that solves a similar problem. He tries to
use it, and it doesn't quite work for him. He could fix the abstraction, but
that would require changing code where it's used. He doesn't want to touch
someone else's code because he just wants to solve his problem. Or perhaps he
just didn't realize how to use it correctly (because it wasn't well designed
in the first place). So he goes ahead and creates a copy of the abstraction,
modifies it slightly to fit his needs and voila - his problem is solved.

Couple of more iterations like this and you get poorly designed code being
copied all over the place with slight variations. If there is a critical flaw
discovered in all these copies, it becomes a major pain to fix. So naturally
nobody wants to touch such code with a ten foot pole. In the mean time, the
bad code gets duplicated more and more like cancer.

------
Ultimatt
In startups a lot of technical debt is more organisational debt and lack of
role specific hiring and poor decisions at a higher level early on when you do
have to move quickly. For every "full stack engineer" you want a real systems
architect, a technical writer, a testing manager/writer, a db
administrator/architect, a UX designer. Plus at least one person for serious
taking care of early stage devops/sysadmin/IT so that whatever you have isn't
fully evolved out of the side 5 minutes someone gives it. You don't want to
give the most agency, to the most opinionated and outspoken developer, but the
least experienced. Especially if they argue for heavy handed engineering
solutions they think are "correct" and this is almost the first ever real job
they have had post university. But everything they say sounds great, and its
the "right" thing you read online so they must be a good engineer, unlike
these other silently working peons. You almost always end up with something
brittle everyone else has to work around, leaving a pile of adhoc architecture
and code that cannot be reused. The original hotshot "architect" you promoted
above the others then gets annoyed and loses a sense of ownership because
their "vision" has not been met by all of these other insufficient "full
stacks". This is the recipe of every toxic BS startup out there. The solutions
are simple, know what the fuck you are doing building a team and dealing with
software. Have people who are head of software who have done more than a
masters in writing a ToDo app in Java one time.

------
tmaly
I have seen some very bad technical debt. Everyone talks about rewriting it,
but what ends up happening is the company will just throw more people on it.

In a more perfect scenario, something like TDD would have been used and there
would be a large test suite. There would also be some really nice well
maintained requirements and technical specifications. Under this scenario it
would be much easier to redesign or refactor the system. But given the reality
of business, this is never the case.

------
arminiusreturns
As a senior sysadmin who has frequently been in the role of CTO because
businesses often don't even have a CTO... my number one pain was always when
the technical debt requires more personnel in order to manage properly but
upper-management refuses to hire more people for the role. I know your
question seems more oriented towards the SV-startup/dev cycle, so my answer is
more for general business (even ones without programmers).

My pet peeve though has always been structured cabling. I don't know if yall
understand just how bad the cabling situation is is many businesses, but it's
bad and for some reason they really don't like spending money on that
particular thing even though it's one of the most important things for keeping
a business up with the times (and the data rates that match).

So, in general, it has been a frustration with the communication gap between
the technical team and the C-level and the board. Too often you either get a
CTO who wants to "program with you" or one who is too much MBA and not enough
tech.

As a sysadmin I have realized my number one failure was not working on my MBA-
side more and thinking purely technical proposals and solutions would win the
day. AKA I should have spent more time golfing and going to lunch so I had
more power with the Cs.

------
lbriner
There are lots of reasons that TD happens but regardless of what they are, the
easiest way to deal with it is to be open with each other. If the COO made a
bad call on something, tell the technical team and work out the best way
forwards. Same if a Contractor wasn't as good as you thought or a new
framework promised lots but wasn't actually good at some of the detail.

When you can be honest about mistakes, tech people are generally more
pragmatic than non-techies might assume.

In the other direction, I clearly communicate the "cost" of short deadlines or
the need to finish something that is in a bad state so that the decision can
be made with eyes open. If a quick fix now leaves us in a poor state to add
new features then the CEO or whoever can accept that so that when they ask for
something new, "sorry, system is too brittle".

In reality, there are many reasons things get rewritten so I also try and
assure my devs that we don't live with these things forever, eventualy we
remove the problems and create a load of new ones! I've rewritten our basic
system 3 times in 5 years and only one of those was mainly down to tech debt
(although we took the opportunity to update to latest tech etc.)

------
goalieca
As a developer, I see Sprint ends as an artificial deadline that people end up
strictly meeting to the bare minimum. I find technical debt accrues most
quickly with smaller stories.

Agile coaches will blast you for estimating workload too high and blame you
for technical debt and rushing. They will say we need to refactor as we go
along. And that stories will need to be even smaller.

Quite frankly, bigger changes are easier to review as a whole and there’s less
wasted time updating the test automation after each small change. Personally,
I like to take a few days to review a change and I like to review a whole
feature at once to see how it all fits together. It can be hard to see that
until all the pieces are ready. Polishing it up before merging is also nice.

In the business world we are led by timelines and milestones and competitor
products. We have fixed dates we need to get things out by. Ive not seen a
business ever run on the agile principal of “take your time and release when
ready and don’t commit to dates but commit to features blah blah”. And I’ve
heard from a few coaches now that the whole organization needs to be agile for
it to work because the business can’t be waterfall while engineering is agile.

Small agile rant aside..

I see technical debt as natural and acceues because we are given limited time
to do something that keeps our paycheques coming. It’s normal for engineering
to whine and for managers to push back. This conflict keeps both sides in
check. Reality would stink if either side dominated the business. So don’t
feel bad about telling your engineers “no”, but also give them enough
independance to manage themselves as responsible professionals. Find that
balance point.

------
yausername
Not a CTO, but in an architect role and spend my life cleaning up tech debt or
building guard rails so it's harder to accumulate.

The most important thing is to teach decision makers (be they product or other
senior engineers) that tech debt is a downstream result of the decisions they
make. That's as simple as keeping them accountable for their actions instead
of hiding the mess under the rug.

Example:

If a colleague uses the word 'later' in a product discussion, pretend they
said 'never' and speak up if it sounds weird.

"We'll get the new feature working with our most popular product areas and
circle back for the others later."

"The feature is ready on time, we'll fix those P3 bugs later."

If you don't call out the use of later->never here, they'll get away with it
and you'll accrue tech debt for no reason. If people want to defer part of
their implementation plan, that decision needs to be on the books somewhere
with their name next to it. If management doesn't see it, it isn't happening.

My most frustrating problem with technical debt is how difficult it is to
shoehorn this bookkeeping into whatever process management or task tracking
system the company uses.

~~~
hyperpape
> If a colleague uses the word 'later' in a product discussion, pretend they
> said 'never' and speak up if it sounds weird.

I can't decide if this is really clever, or really misguided. Often "later"
does mean "never", and sometimes that's ok, sometimes it's bad.

However, you can't do everything at once, and there are plenty of things that
you really will do later, because you know that as soon as you implement one
feature, a customer will request the logical followup, and you will do it.

------
bigpicture
I can't answer as a CTO-like person, but I'm in an Architect-type role and
deal with our CTO-like person on a daily basis.

The high-level management cares only about "The Big Rewrite" because it is the
only thing that gets visibility at that level of the organization and it comes
with a large budget that everyone knows will be replenished for years as it
gets spent. Our CTO-like person has fully bought into this idea.

We have too many projects and too few developers, so incurring technical debt
is an everyday thing. It will be dealt with in "Phase 2" of the project, which
all the developers know is something that never happens. From my vantage point
- having one foot on the developer side and one foot on the management side, I
see that the management thinks the developers are incompetent and the
developers think the management is incompetent.

It's not a good position to be in, yet at the same time we have a pretty laid
back culture, work less than 40 hours per week, great benefits, etc. So
turnover is fairly low. Most of the developers have either a side gig or are
going to school part time, or else they'd probably go insane.

------
taeric
Ironically to me, the most apt application of "debt" to any source you do is
not the hack or shortcut code, but the libraries you bring in. You are
literally borrowing against the knowledge and expertise of someone else.

The _vast_ majority of the time, this is the correct thing to do. But _if_ you
find yourself scaling to absurd levels, this is the code you will have to
worry about. You will find that you almost certainly will start to care about
all of the details of the network that you were able to ignore in the past.
Connection and thread pools you were able to ignore, now need you to invest
directly in.

The hacks? Just make sure you aren't worried about aesthetic changes. Keep
people on point solving problems, but don't berate them for wanting things to
look at certain way. Developer happiness is important. Often that rewrite of a
module that you thought would take too long, can help morale so that folks are
moving faster by the end of it. Just make sure to keep it goal oriented and
for the love of all things good, don't let them get away without keeping
feature parity for your customers.

------
pjbster
Not a CTO and I've never come close to having a conversation with one. As a
developer, though, I've been burned by tech debt too many times to feel
comfortable with it.

The biggest problem with technical debt is that everyone has a different
tolerance for it. Some even have an appetite for it. This is a breeding ground
for conflict.

The next biggest problem is that the people who take it on are rarely the same
people who end up paying it off. The former tend to get showered with kudos
whilst the latter experience stress and poor performance reviews.

IMO the worst kind of technical debt is that which is taken on to route around
a business process ("Governance"). So it, sort of, contributes to faster
delivery but the conditions which drive the choice tend to persist until a
regime change so the compromise sticks around forever.

In an ideal world, you should only agree to take on technical debt if you are
also presented with a repayment plan. And that leads to my final problem with
technical debt: most people tend to assume that that last part will take care
of itself.

~~~
pavel_lishin
> _IMO the worst kind of technical debt is that which is taken on to route
> around a business process ( "Governance"). So it, sort of, contributes to
> faster delivery but the conditions which drive the choice tend to persist
> until a regime change so the compromise sticks around forever._

Can you give an example?

~~~
pjbster
Client access to database: too much effort to get stakeholder sign off for api
changes and regression testing so we'll just grant the client full access to
the database schema. Now we can't change the schema without breaking zero, one
or many clients.

DBA deployment processes: insist on taking system offline for cold backup.
Every. Single. Time. Solution: make application run dynamic code which is
stored in a database table. Effect releases by issuing UPDATE statements. It's
hard to argue that an Oracle database can't handle one of these reliably so
the DBAs don't require cold backups for these. Still, we no longer have
compiler support so our tests had better be solid. Bonus con: the DBAs
cottoned on and started doing the cold backup thing for these updates so we
lost business agility too.

Finally: a legacy web service. No toolchain but we do have WSDL and source.
Ported to Weblogic in 30 minutes. Fits in with current infrastructure, more
performant and gives company an opportunity to decommission the old box. But
PM hasn't got a Jira for this work and we're already into integration testing
so throws it out. The architect resigns.

Governance. A.k.a. Insurance-Against-Corporate-Liability. Responsible for at
least 3 fucked up systems during my tenure there.

------
svilen_dobrev
IMHexperience... There's no complete escape from technical debt, there's only
reshaping / transforming and moving it elsewhere where it does not hurt (well,
defects/bugs do not count as technical debt).

In most cases this fix is by rising complexity - as number of (meta-)levels of
the system (mind you, that is always humans+machines system, stretched in
time). So what is being used everyday by everybody becomes a piece of cake..
and noone is to touch the magic interpreter that lets it happen. And funny(or
sad), that magic thing could also be done perfectly by itself.. so it looks
like there's no technical debt... Just the experience-power needed to
understand it, is like 100x what is needed for the easy stuff it has made
possible. So, essentialy, technical debt has turned into "experience" debt. a
Futures contract of a kind... or maybe i'm just too pessimistic..

------
ryanSrich
Tech debt, whether creating it or paying it is a critical function of the
business. Where this gets messy is when an engineering organization feels they
own or somehow should prioritize the debt. This is especially true of paying
it.

In every case that I've been involved in or had explicit oversight in creating
tech debt has been when customer and revenue obligations are needed to be met.
It's very rare in startups to have artificial or made up deadlines. I'm sure
they exist at larger orgs, but as a startup executive, I can say with
confidence that everything is constantly on the line, each day is critical,
and so shipping product is paramount.

Of course, you have to pay that tech debt back at some point, but this is a
business decision. It may turn out the business needs to kill that product or
alter it in such a way that paying back debt is useless.

------
corpMaverick
Technical debt should be paid one commit at a time.

You cannot pay it at once. Just make many small continuous improvements.

------
nichochar
Slight tangent, but I personally don't like the term debt.

It's an attractive analogy but it doesn't go very far; worse, it leads people
to think of it as easy to reason about.

The biggest issue by far around tech debt, which has been covered thoroughly
throughout this thread, is that measuring it is tricky.

Measuring financial debt is trivial, you can put a dollar amount on it.
Measuring tech debt is very subjective, as it depends on the level of
seniority of your team, their ability/authorization to make changes (in
banking, you can change nothing, in a startup, you can change everything),
attrition rate, etc...

It's such a complicated equation, that using the term debt tricks upper
management (non technical) to think it's easy to predict: it really isn't at
all.

------
bytematic
As the team gets larger AND/OR the product attains its necessary features,
"technical debt" becomes more of a problem. I would love to address it at
every stage of a product, but it is simply not possible, many times during
early development our team is just figuring things out, seeing what works and
what doesn't. If we were to analyze future development for each iteration
during this stage, it would be a nightmare. I like to trust in my team that
they have a good understanding of best practice and choosing between tough
work and future work. When the product "feels" at a good stage, say a few bug
fixes after release, we spend much more time refactoring.

------
lumost
Personally I find that the term tech debt gets overused. I've often heard it
applied to relatively fresh projects for features/projects that just haven't
been done yet, or to pitch the architecture of the day.

The times when I have seen a real breakdown where the term tech debt is
legitimately used is when a scrum process breaks down deliverables below their
smallest atom. E.g. different stories to write the code and unit tests,
projects spread across devs such that no individual has ownership of any
portion of the product etc.

In such situations we're not talking about debt as much as we're talking about
junk.

~~~
ghiculescu
Yeah I always laugh when there’s a project with no code written yet, but with
plans for how we will “pay off the debt” that will inevitably be created.

It doesn’t need to be like that if you run your projects well.

------
pacoverdi
Technical debt is code that makes everybody waste time (eg. because of its
complexity), or that creates problems in production (eg. performance issues)
BUT that you cannot easily replace because it is too tightly knit to the rest
of the architecture, or it would be very long/risky. So you have to live with
and pay its dividends every year.

The most frustrating thing for me is that you cannot throw it away and start
from scratch (if you want to keep your users). You have to bite the bullet and
find ways to make that ugly/smelly code go away in very small increments. This
process can take years.

------
Feeble
I think it is important to understand that technical debt is very difficult to
define and quantify. Depending on your role, responsibility and skillset you
will consider technical debt to be different things, and often attribute
different (negative) impact of it.

Technical debt can cripple a development team, but I have also seen it being
used as a continuous argument to invest into system development that really
had zero positive business impact at the end. I am not saying to ignore it,
but I also know that as a developer you tend to overvalue the things that are
just in front of you.

------
fuhrysteve
I find that it doesn't get too frustrating if you don't let it get out of
hand.

Routinely prioritize keeping dependencies up to date, clearly depreciate "old"
/ bad code, and focus on coming up with good product designs that have
reasonable compromises between time-to-market and high quality engineering.

You just can't let stuff slip. Don't let 6 months go by without at least
_trying_ to upgrade to the new version of whatever or without making some
_some_ progress moving from the "old way" to the "new way" for some component.

------
bnchrch
I believe people are mis-framing the tech debt debate or at least missing part
of it.

Tech debt has to be managed and triaged, this is true. Tech debt slows down
product development and degrades stability, also true.

But tech debt is also a talent retaining and hiring tool. No one enjoys to
work on a project that gets in your way for reasons that can be resolved. You
need to pay down tech debt when people ask in a reasonable fashion otherwise,
simply put, they'll leave.

------
jaryd
As a follow up, what are the most useful methods for managing technical debt
without incurring too high of a cost to team/product/project velocity?

~~~
fgheorghe
Address issues as they are raised. Don't wait too long to get rid of debt, and
most certainly don't rely on fixed time cycles of fixing things. A team has
good sense of the issues they deal with, and you can capture their view in
retrospective meetings. Depending on how bad the debt is, you should allocate
time to improve. Velocity will only get worse as debt grows, so there is a
clear business case for such stories: the more debt, the slower the velocity
and the lower the quality. Spend time now, to speed things up. But just don't
do it from a theoretical perspective. Address real issues, set real goals and
measure them.

------
fgheorghe
"And you know this rewrite, like others in the past, will be a significant
sink of time, and likely a catastrophe."

To avoid this establish clear goals for the rewrite phase and measure results.
Don't just rewrite for the sake of rewriting. Have a session where you
identify issues and how to tackle them.

From a CTO-Like perspective, getting rid of technical debt should happen
frequently, while keeping business goals in mind.

------
byebyetech
I think technical debt should be called "management debt" because management
is the primary source of such debts. Technical debt implies it something
engineers forgot to do or were just lazy to do it. While it maybe the case in
some places , mostly its Agile/Project planning that lacks any appropriate
place for fixing technical debt and refactoring.

------
tlynchpin
I never liked this term because debt as we commonly relate to it, especially
in business, is a quantifiable obligation that can be discharged.

In contrast "tech debt" is not a thing with a quantity, no one can know when
or even if it will come due.

------
taude
Legacy unsupported open source projects with outdated dependencies. (We work
on a very large Java application (1.8M lOC) and were acquired by large
financial institution that has a lot of policy around open source libraries
and support.)

------
baybal2
Not a CTO.

1\. Not only legacy code eats resources that can be thrown on new
developments. While legacy code is most troublesome, it still will be at
maximum only twice as bad resource-wise, than new and pretty code.

2\. As company grows, it may well be that a piece of software being used by by
less than 5 people at the company, but require an own developer team to
continue actively support it.

3\. Maintenance resources vs resources spend on development is a never ending
drama.

4\. You should proactively think how to cull you internal projects, and be
able to dictate to the rest of the company to stop using it.

5\. Custom software for "business guys" \- an easy way to multiply IT support
budget. Hiring technically illiterate analysts, finance people, supply chain
people costs a lot, a great great lot.

6\. Last point. THERE SHOULD BE AN IRON WALL in between people making "money
making code" and all other tech at the company. Can't stress this more.

~~~
frant-hartm
Could you elaborate on the last point?

~~~
baybal2
Well, the most extreme case will be for example: in an online ads company,
somebody disrupts the team responsible for bidding software to do somebody's
pet project, do a UI job, or making them do software for internal IT needs.

------
_Codemonkeyism
A point I often make about technical debt:

[http://codemonkeyism.com/ceo-to-cto-what-is-your-
rewriterati...](http://codemonkeyism.com/ceo-to-cto-what-is-your-
rewriteratio/)

------
ppeetteerr
By biggest frustration with technical debt is that, at some point, it becomes
so great, that the developers are more busy maintaining the existing code than
they are building new, business critical functionality.

------
rmoskal
Given a software product with a modicum of market fit and or an established
user community, AND a competent engineering organization, there is no reason
to have technical debt in 2018.

------
legohead
Nine times out of ten the issues are being exaggerated and can simply be
ignored. That's not what the engineers want to hear of course. They are eager
to fix things, to stop their daily grind.

It's very rare to actually get stuck due to technical debt. The times I've
seen it happen, we grabbed all the (relevant) developers, came up with a
solution, and fixed it quickly.

As long as you don't have high turnover, technical debt is fine. You have a
team who is aware of all the quirks of the codebase(s) and knows how to deal
with it.

~~~
pavel_lishin
> _As long as you don 't have high turnover, technical debt is fine._

"If you tell them to shut up enough times, eventually they will."

> _They are eager to fix things, to stop their daily grind._

Yes, because being ground down daily will eventually wear you down.

> _You have a team who is aware of all the quirks of the codebase(s) and knows
> how to deal with it._

But that's not a good thing. If your technical debt adds an hour to a dev's
day, every day, you're running at ~85% capacity. If you have tests that need
to be babysat, or code that takes a long time to add new features to (or fix
bugs in) because of tech debt, you're losing valuable time.

Even if it's not necessarily the sort of thing that brings a codebase to its
knees, or the kind of thing that sinks a business, but it is the equivalent of
refusing to tie your shoes and tripping and falling as you keep trying to walk
to your destination.

~~~
SmirkingRevenge
> Yes, because being ground down daily will eventually wear you down.

This is true. If too much technical debt accrues unchecked over time, it can
have a real effect on the quality of life and morale of your engineering team
- which eventually manifests as high turnover, which in turn leads to more
tech debt, in a vicious feedback loop.

I've worked at such a place, and the only relief from that cycle came in the
form of lost clients. But then again, despite death spiral, the company was
bought out, and the owners made out pretty well.

I think maybe the old adage about outrunning a cheetah is apt here... to
outrun one, you only have to run faster than the guy next to you.

~~~
pavel_lishin
> _which eventually manifests as high turnover_

Or, arguably worse, a crew of beaten-down 9-5 lifers who don't particularly
care about their job.

------
adreamingsoul
I question the motive to complain about tech debt with the “solution” already.
For example, microservices will fix it.

------
lkrubner
" _And you know this rewrite, like others in the past, will be a significant
sink of time, and likely a catastrophe._ "

This statement can only be true if you're using a very specific definition of
"microservices". The first time I wrote about small services, in 2013 [1], I
had not heard the phrase "microservices" so I used the phrase "an architecture
of small apps". I was thinking of apps that talked to each via Redis, or HTTP,
or didn't talk to each other and pushed everything into a database -- my idea
was broad. Then in March of 2014, Martin Fowler wrote his essay on
Microservices [2], which brought the phrase into widespread use. Soon after
that there arose a number of startups that were promoting microservices, and
they were promoting a very specific ideal: a system of apps that spoke to each
other using HTTP, finding each other via auto-discovery, and offering data via
a RESTful interface, with Oauth for authentication. A large number of people
then began to use "microservices" for this specific set of interlocking ideas.
That's fine, but it is a bit rigid. Something so comprehensive can demand a
full re-write. But there is the more general idea, of slowly breaking up a
monolith, and replacing small pieces with independent apps -- that does not
demand a full re-write. That can be done slowly, and you have the option to
keep the original monolith in a limited role, often to handle the frontend.
Martin Fowler said all this in his essay, but some of the more extreme
proponents of the idea have ignored the nuances that people like Fowler or
myself have written about. I myself used the phrase "microservices" for a few
years and now I've given up on it as too many people are associating with a
very specific set of ideas, instead of the general idea of small apps
cooperating with each other.

All the same, the idea can be powerful. Check out "Two months early. 300k
under budget" [3]

[1] [http://www.smashcompany.com/technology/an-architecture-of-
sm...](http://www.smashcompany.com/technology/an-architecture-of-small-apps)

[2]
[https://martinfowler.com/articles/microservices.html](https://martinfowler.com/articles/microservices.html)

[3] [http://thoughtworks.github.io/p2/issue09/two-months-
early/](http://thoughtworks.github.io/p2/issue09/two-months-early/)

------
jerf
As with others, not a CTO, but an architect for many years, and I know the
perspective I'd start out with, which is that the biggest problem at the CTO
level is the mismatch between the people making decisions about what we work
on next, which once you are no longer a 3-5 person start up is not the
developers, and the people who understand where the technical debt is and what
could be done about it.

Transferring the knowledge across that gap is very difficult, perhaps
virtually impossible, because both sides of that transaction have a lot of
incentives to mangle the message in the process.

The manager who is deciding what to do, and likely incentivized one way or
another based on features, uptake, performance, etc. is going to always be
very biased in favor of providing more to the customer, and will want the
developers always visibly working on something that provides visibly more to
the customers. They don't want to hear about why we need to rewrite this thing
for no customer-visible change except maybe "fewer bugs", and probably don't
want to hear about why the project is slower than expected ("just fix it! but
without any additional resources") or what could speed it up.

The engineers have even more complicated problems. First, the engineer may not
be able to correctly determine what technical debt even _matters_ if they are
not properly kept in the loop about where the product is going. That is, bad
code on a feature we're about to dump or obsolete is way less important than
on a core feature. Keeping engineers in the loop about future product
directions seems to be a big challenge. (I mean, I've seen it done
successfully, but it takes a lot of effort. It's just so easy for the manager
to go to some four hour meeting with the CEO and the board about the product
and all that ever gets back to the team is "We need feature #64." I mean, OK,
great for distilling those 4 hours down to the next action item, but at least
your senior engineers really could use a bit more guidance about where the
product is going.) Second, the engineer is likely to have a personal
connection to the code, which can have any number of effects ranging from
being defensive about any suggestion that the code is less than perfect, to
believing that everything, everywhere is "technical debt" and really needs to
be rewritten.

To properly gauge the technical debt on a project takes an almost perfect
merger of both the business side of the project and full engineering knowledge
of the project. It's harder than even spec'ing out a new feature by far. I
don't know that I've ever seen a great solution for this; the requisite bits
of knowledge are structurally very hard to bring together.

(This is written from the assumption that technical debt is in fact a big
problem. There's a distinct possibility that it really isn't, but I'm
answering the question as written. In general if I was starting a new CTO job
at anything but a very established software company, I would be surprised and
pleased if "technical debt" really was my biggest problem, and not flawed
processes, broken cross-team communication, or goodness forbid but all too
likely, _internal_ team communication.)

------
wellpast
The idea of Technical Debt, though prevalent, is fallacious, because implicit
in the idea is the belief that time-to-delivery is somehow _necessarily_ a
function of code quality.

But this is not true. Here's why:

The first thing to realize is that code quality is an objective quality; it is
the degree to which your code is decoupled, the narrowness of the interfaces
between components and the relative size of each component. (This is objective
because you can take two systems that deliver the exact same business function
and compare them wrt their degree of coupling, cohesion, and interface width.)
The first fallacy is to think that code quality is somehow subjective; it is
not: not at least not for the sense of code quality that matters--which is how
much _necessary_ business cost there is to evolve the code. (Note that
_necessary_ is the key word; obviously it will always cost an
inexperienced/novice a lot of time to contend with a code base regardless of
its structural qualities.)

Another thing that gets conflated when talking about Technical Debt is the
"flexibility" or generality of the system that is built. It is a mistake to
consider a specific (ie, non-generalized) system as having higher technical
debt. That would be like saying Gmail has high technical debt because it
doesn't have Kanban features. A high-quality systems (low technical debt, so
to speak) can be a system that is quite specific, quite hard-coded, etc.

These are two fallacies. Now that they have been called out, the last thing to
consider is whether or not there is some _necessary_ cost savings in
delivering a system with "higher debt" (ie, a system with more complexity).

As an experienced practitioner who has worked with many other experienced
practitioners in large and small companies alike, I can say confidently that
one's ability to deliver a simple system is a function of skill set not time.
In other words, a skilled practitioner will take a no extra time to deliver a
simple system albeit perhaps a specific one. There is a necessary time trade
off if we are talking about making a system with more generalized capabilities
--but at this point we are talking about features and priority and managing
feature creep and the like.

This is where I think many startups fail when they think that 5 mid-to-junior
engineers are better than 1 qualified* senior. The only caveat is that there
are many (and, perhaps, most) "seniors" out there who have a not all
cultivated the relevant skill set. Who have spent their careers leaning on
this false notion of Technical Debt and the idea that "I simply didn't have
the time to make a good system"\--in this line of thinking, there will be no
growth because implicit in it is the idea that there really is nothing new, no
now skill set, to learn, that all there is just a shortage of time.

A tennis player would never think, "I would played a better game had I had
more time." As if somehow there is a world in which tennis could be played in
slow motion. No: the tennis player practices and practices until her muscle
memory is tuned to play a competitive game in real time.

Likewise, a developer must _specifically_ train herself to play a good game in
real time. So that the ability to deliver quality is part of their muscle
memory. At that point it is actually harder -- takes longer! -- for this
developer to build a complex system. Decoupling becomes instinct. Much like a
skilled tennis player would have to go out of their way to play badly, with
bad form, etc.

Unfortunately I don't see this training happening in either academia or in the
practitioner's world. Everything is too optimized for immediate delivery and
so people spend their careers learning how to hack it through and then only
_after the fact_ come up with this idea that Technical Debt is somehow a
necessary thing. The idea of Technical Debt is alluring because it is much
easier to lean on than the idea that you have a serious lacuna in your skill
set.

Now one might say I'm not being convincing because you still have to take the
time to "learn" the skill set. This is true. But this is not change the above
reasoning at all. Unless one thinks that learning and delivering can happen
together in some optimal fashion. I don't believe that they can. I do not want
the person building my home to be learning on the job, that is for sure.

------
jsd1982
As a dev, "or solve it with microservices altogether" hahaha. Good one!

------
lowry
I've been CTO (well, IT Director) for 3 years and I saw no particular problem
with technical debt. I just handled it the same way I would handle financial
debt. Repaying during good times and taking on debt in projects with high
stakes and fixed deadlines.

It all depends on the industry as well. My CTO experience was in News and
Media, where software is rewritten quite often and does not have any business
value by itself.

I am now in a consulting position in the banking industry. Things are
different there. None is willing to take on the risks associated with
technical debt repayment. It accumulates endlessly. Once it reaches a point
where no local developer would work on the project, the project is handed over
to Asian sweatshops which support it indefinitely.

~~~
waterlink
> Repaying during good times and taking on debt in projects with high stakes
> and fixed deadlines.

I agree this is an amazing strategy when you can do it!

> None is willing to take on the risks associated with technical debt
> repayment.

What do you think they are afraid of? What risks are there? Is it possible to
mitigate these risks in some way?

~~~
carlmr
>What do you think they are afraid of? What risks are there? Is it possible to
mitigate these risks in some way?

Not OP, but usually there are no (trusted) unit tests.

~~~
lowry
IMO, unit tests, integration tests, TDD, BDD, etc do not matter as much as the
size of the codebase. If your system has millions of lines of code, you are
doomed and no amount of unit tests can help you.

~~~
hutzlibu
I believe thats why modularization was invented?

~~~
lowry
Modularization per se does not work. What works is administrative boundaries.
Once a product reach a certain size, the team should be split up.

~~~
hutzlibu
"Once a product reach a certain size, the team should be split up."

That is also called modularization.

------
shp0ngle
Refactor everything all the time, duh

