
The Human Cost of Tech Debt - BerislavLopac
http://www.daedtech.com/human-cost-tech-debt/
======
ThrowMeAway314
I recently became CTO of a company with two moderately successful SaaS
products. The second one is a fork of the first, but has been maintained by
more competent people than the original product. Both are still monuments to
technical debt.

About 4 million lines of PHP code, written by underpaid, sometimes not well
meaning, freelancers and students over the span of 8 years. The CEO wrote a
large part, but stopped learning new techniques around 2004.

I'm bringing competent and well paid people in through my network and try my
very best to give them as much freedom as possible, I allow and encourage
greenfield modules/services that run on separate and new infrastructure for
anything that is possible to be rewritten in the timeframes, but the larger
part of the job is still mind numbing to my team and that makes me question my
wisdom.

If anyone has tips on how to steer such a ship in a direction where the work
is less frustrating for my devs I'm very open for advice.

~~~
foz
I've run across this situation many times (I'm a "senior" team lead, meaning,
I've been working for 25+ years). I've witnessed companies that overcame tech
debt, and seen companies fail because of it. There's basically three
approaches that people can take.

One is the "big re-write". They start on a new code base, and try to develop
it in parallel. It takes a very long time, and the teams have to work on two
solutions for some time. It's a big bang approach, and it often fails, or
drags on for years.

The second is massive refactoring. It requires intensive testing and best
practices. This strategy requires that the teams focus on testing intensely.
However, often the testing culture is not there, which is why the code became
unmanageable in the first place. It's kind of like starting over. And the new
focus and discipline on testing is hard for teams to do without strong
leadership, training, or new talent.

The last, and most effective in my opinion, is to go with a service-based,
incremental approach. If the code base is not already using services, APIs
must be built. Frontend/apps must be de-coupled from the legacy components. A
clear domain model has to be agreed upon, and then parts of the legacy
codebase are put behind APIs, and de-coupled from the rest. Over time,
sections are refactored independently, and the APIs can hide legacy away.
Maybe the legacy parts are refactored and replaced, or they stay around for a
while. But the key is, that this approach allows multiple people or teams to
work in parallel and focus on their areas. This is domain-driven design in
action, and it works. New features can actually be developed sooner, even
though the legacy is not replaced yet.

In the end, overcoming tech debt is about people. And on larger code bases,
it's more of an organizational problem than a code problem. Developers need to
be able to move forward without having to navigate too much code or too many
different people.

~~~
0xfaded
Do you subscribe to the microservice philosophy? I ask because you come from
the era of shared objects, and I personally still consider libraries with well
defined APIs to be much simpler than dealing with multiple processes possibly
running across different hardware.

I do break my infrastructure apart, but far less aggressively than some
advocates.

I'm curious what 25 years had lead you to believe.

~~~
jaegerpicker
I'm not the op and I don't have 25 years experience but I'm a
manager/architect with 18 year's behind me. My take:

I very much believe in microservices. I've repeatedly seen small library based
approaches fail because one of the key and near universal truths of tech debt
IME is code being tightly coupled. When you force an API interaction to happen
via an outside protocol, you force a clean contract and a culture of coding to
a contract. Decoupling the code allows yet team to move faster and more
independently.

------
blunte
"They know that they’re going to have to manufacture endless explanations for
why seemingly simple things take them a long time." This is what kills me. You
can't say to the boss, "Your beloved senior dev built this arcane and fragile
system, so everything I do takes forever." Instead you have to find
diplomatic/meaningless explanations for why you're moving so slowly.

~~~
bradleyjg
From the bosses' point of view it is very difficult to tell the difference
between "we have a crap codebase" and "the new guy sucks at reading code and
is going to want to rewrite everything he touches".

~~~
whack
If 5 different people are all saying the same thing, that makes it a lot
easier.

~~~
marcosdumay
It's not rare that a company hires 5 bad developers (or more) for each 1 good
one.

------
simula67
I wonder if it is also the case that if the code quality is high, people are
happy and they stay. This means fewer openings are available in those teams.
Therefore, if you take a new job, it is more probable that the code is bad and
there is a revolving door of developers who have tried to make it better,
failed, left and created an opening for you. The vicious cycle of bad jobs.

~~~
whack
Honestly, I would get bored working at a place where everything is "easy". I
actually enjoy the challenge of working on awful code-bases, and trying to
improve upon it, while still preserving the functionality and not breaking
anything.

That said, if I had to work with awful code _and_ I wasn't allowed to touch it
or make it better, I would definitely leave in a heartbeat.

~~~
OpenDrapery
I used to enjoy the thrill of the refactor, especially on applications where
there weren't a lot of users or the product didn't generate much revenue. Or
better yet, refactoring before the product has even been released. When the
stakes are low, you can be much more fearless, and it is fun to rearchitect
and introduce new patterns, etc.

But when the stakes are higher, the risk/reward ratio just isn't there. You
can refactor the code to make it more sensible or pleasing to your eye. Two
problems with this: 1) Someone else may disagree and refactor it again. 2)
When it blows up or has unintended consequences, and you are now on the hot
seat, you find yourself working on Saturday asking yourself if it was worth it
at all. Thus the fear sets in.

------
jcbeard
Bad management, oversight, and architecting returns bad code. Often team leads
know better, but are driven by deadlines and the lack of a skilled workforce
(with little time to train them) to produce a product. Of course this leads to
turn-over...eventually. People will work for a year, realize that nothing they
want to accomplish will happen quickly because of resourcing, then they'll run
away. I really don't agree that the human cost of technical debt is a new
concept. It's foremost in my mind when I think about giving tools to employees
to add new features. Well written article, but not really anything new.

------
flukus
Stuff like this is why I wish you could review the companies code before
joining. Getting hired, discovering how bad everything is and then starting
the job hunt again is awful for everyone.

~~~
brandall10
For most job offers I've gotten I've asked to review the source code post-
offer. I've even done this in a couple interviews -- can't recall ever being
denied.

In one particular instance I came back to review the code for a product that
required a level 3 security clearance; the hiring manager happily complied,
letting me spend over 2 hours alone studying the code on a workstation in a
side office. One reason I passed on that position was due to the code being
more spaghetti than I would have preferred.

~~~
iamcreasy
That's an excellent idea. How to ask about it cordially?

~~~
brandall10
Inside of an interview, what usually happens is we get into a discussion of
the architecture of the product I'd be working on. Then I pepper questions
about the codebase itself, and after a few questions I'll say "Wow, that
sounds interesting. Mind if I take a look?"

Post interview I'll talk to the hiring manager and express how much I enjoyed
meeting with the team and love the product, then ask if it would be alright to
stop by to take a closer look at the codebase, watch how the team works in
action, just be a fly on the wall.

------
caente
Copying my comment from the OP

> Analysts and project managers might account for technical debt when
> discussing slipped deadlines.

No they won't, you cannot quantify technical debt, it's a _metaphor_.

I completely agree with the rest of the post. But "technical debt" is not a
cause, nor a valid metaphor to share with management. Other bad metaphors are
"building" and "architecture". We are not building anything, we are only
writing algorithms that computers will follow. When we write down those
algorithms, we are also encoding the inner workings of the company. The true
knowledge of how a company works is not only in its documentation or in the
heads of their employees, it's also in its automated processes.

The automated processes are the reason why a company can be competitive
nowadays. People need to be able to understand those processes, those
algorithms, _from the code_ , documentation is never enough, and is never up
to date. The code is the primary source of information. If it's not legible,
if it becomes arcane knowledge, a black magic that only a consultant can
"fix", then the company grinds to a halt, it withers and is crushed by
competitors. Be careless about the code quality is not about the suffering of
the developers, it means that you are burying vital knowledge of your company,
and eventually no employee, no consultant, will be able to dig it out.

If we are trying to figure out how to deal with managers who don't care about
the company, but only about absurd deadlines that their managers gave them,
then we need to start pushing back, start saying NO. Politics are unavoidable,
and often necessary, not all decisions can be made because technical reasons,
but we need to start imposing the reality of the code, you cannot rush it, our
you are killing yourself.

------
ddebernardy
> For a manager, a code base high in technical debt means that feature
> delivery slows to a crawl, which creates a lot of frustration and awkward
> moments in conversation about business capability. For a developer, this
> frustration is even more acute. Nobody likes working with a significant
> handicap and being unproductive day after day, and that is exactly what this
> sort of codebase means for developers.

The very same holds for managers and teams in companies that need to deal with
organizational debt:

[https://steveblank.com/2015/05/19/organizational-debt-is-
lik...](https://steveblank.com/2015/05/19/organizational-debt-is-like-
technical-debt-but-worse/)

------
FollowSteph3
The concept of technical debt is easy to convey to management, the hard part
is conveying the scale of the debt because it's an abstract value. You don't
have x units of debt. And because there's no actual value to measure it's easy
to keep adding it on, it's easily viewed as the same debt.

To give you an analogy imagine you have a credit card. As you spend you
increase your debt but you never get to know want your balance is. All you
know is how much you pay per month. In fact even that number would be blurry
and fluctuate. That's what management sees from their perspective. As a result
it's easy to keep adding debt because it had no real value, you don't see that
you're $1000, $10k, $100k, etc in debt, just that you have to pay something
each month. Yes each month you have less to spend but it's a lot less than you
spent that month. Last month you had to pay $100 but this month it's $101 and
you don't have to pay that $50 one Ike hit. The extra $1mth is easier. You
kinda forget that in a year it's now an extra $2-$3/mth. And again you never
see your balance, so you don't know how much you have in debt. Your spouse
keeps saying your way in debt but you have no idea of the scale. It's very
easy to increase your debt this way.

Unfortunately I don't have an answer as to how you can value the technical
debt so that the business people can appreciate the scale. I don't think as
developers we can even measure it ourselves accurately, it's more of a
feeling, a scale if you will.

------
sigsergv
Most articles about technical debt should actually be about _technical
default_. Tech debt is a good technical management instrument, it's just need
to be paid and everyone involved must understand that. Delaying payments is as
bad as delaying regular finance debt payments.

~~~
techterrier
Completely agree with this sentiment. Not all tech debt is bad - and it's
often inevitable. I wrote some stuff about this notion:
[https://medium.com/@MostlyHarmlessD/on-technical-
debt-7bac65...](https://medium.com/@MostlyHarmlessD/on-technical-
debt-7bac65edf349)

------
coderot
This article hits close to home!

What should be done in a company that's basically committed every error you
could commit? What's an engineer to do? Just switch jobs?

I joined a company a few years ago that had hit a wall with its tech stack
after 8 years. They brought in a new technical lead who convinced everyone to
approve a "rewrite everything from scratch on new stack" approach (even though
it rarely works).

2 Years and $40-50m later:

\- The thing's nowhere near done. Customers all still using old legacy product
that's not been updated in 2-3 years now since resources were diverted to
rewrite

\- 88% of the eng org has turned over

\- current eng org doesn't know how 20-40% of the codebase works

\- customers have caught on and started leaving en masse

\- Rest of company has turned over as well as they realized nothing useful was
going to come out of engineering for years more if ever

It's a bit confusing. What the heck should we do at this point?

I take some solace in knowing that I've seen firsthand what NOT to do in
situations like this.

~~~
flukus
> What should be done in a company that's basically committed every error you
> could commit? What's an engineer to do? Just switch jobs?

Sometimes that's all you have the power to do. Why are customers leaving en
masse if the old product is still functioning?

It sounds like the rewrite got stuck in the pattern where it has to do 100% of
the original and then some. Just like a new product, a rewrite has to hit that
MVP mark, just not as urgently. They also have to be able to throw out the
cruft and not import every bad idea from v1.

~~~
MichaelGG
Maybe competitors are providing a better product, so customers leave? It's
hard to make a new version of a product that does less. Features were
presumably mostly added for a reason, and customers probably use them.
Removing them is a hard sell.

~~~
flukus
If there are better alternatives then sticking with the current technical debt
may be a non-option as well.

>Features were presumably mostly added for a reason, and customers probably
use them. Removing them is a hard sell.

Trying to make everyone happy is one of the big drivers of technical debt, a
lot of stuff that gets added just to get that one critical contract. Sometimes
you can refactor to better handle these variations that weren't anticipated,
but sometimes you can't and you have to evaluate the complications versus that
one customers business.

You say this most often in businesses that practice sales team driven
development.

------
boothead
This is a great article. I've also seen tech debt exacerbated by poorly
implemented agile practices. One of the worst projects I worked on kept
kicking the can down the road because of a misguided idea that everything
should be a customer deliverable. So not only were we forced to work with
crap, but we were disempowered from making it any better! It was a nightmare,
motivation and team cohesion was non-existent and yes, turnover was very high!

~~~
clifanatic
> everything should be a customer deliverable

And everything should be delivered in a 100% predictable timeframe and that
predictable timeframe should be about a day.

------
gearhart
> To tie this back to the tech debt metaphor, think of someone with mountains
> of debt trying to explain being harassed by creditors. It’s embarrassing,
> which is, in turn, demoralizing.

Seems that a more apt use of the analogy would be "think of someone with
mountains of debt trying to explain why they can't afford to do the same
things as their colleagues"

------
sigsergv
The human cost _of unpaid_ tech debt.

------
ubercode5
"If you’re not already familiar with the concept of technical debt" ohh how I
envy thee.

------
clifanatic
As much as I love the metaphor, it's been quite a while since I first noticed
the "eye-twitching" among non-technical management types whenever the term
"technical debt" comes up. I can almost hear them thinking, "oh, boy, more
spoiled programmer bullshit for me to deal with".

