
A Field Study on Technical Debt - heidibrayer
http://blog.sei.cmu.edu/post.cfm/field-study-technical-debt-208
======
stevecalifornia
I have never worked in an environment where new employees didn't show up and
declare that the existing infrastructure sucks. Knowing this, I caught myself
doing the same thing. Reflecting on it, for me it was a coping mechanism. I
was overwhelmed by all the new things and didn't understand why this was setup
that way, etc.

The reason I bring this up is I am surprised that architectural choice is
overwhelmingly the top technical debt concern. That doesn't seem right. I
would expect it to be poor code due to time constraints.

~~~
bsder
In most places I have been, the infrastructure _DOES_ suck.

First question: "Backups?" followed by "When was the last time you did a
restore?"

Second question: "What's your revision control?" For all of my slagging on git
and the people who use it, I am _thrilled_ if I hear "CVS" or "Subversion"
because it means _they 're using version control_. "git" or "mercurial" tells
me I have an team that has some level of clue.

Third question: "Build system?" Good luck. Never yet seen it.

~~~
ryandrake
I took a development job once, where I learned on the first day:

1\. No backups. Emergency plan = several older copies of the product's source
spread among hard drives of various former employees' systems.

2\. No source/revision control. Current version of the product's source code
was literally whatever was on the lead developer's hard drive at the moment.

3\. No deployment process. When a release had to happen, above mentioned
developer would build an EXE out of whatever he had (if it built) and directly
deploy it to customers.

4\. No dependency management. What versions of vendor-provided libraries do we
use? Whatever is on the guy's hard drive. Oh, and they're all binaries because
we either lost or can't build the source anymore.

5\. No bug tracker. The list of stuff that needed to get worked on was
whatever the CEO complained about last. There was no concept of a backlog of
existing bugs or technical debt.

6\. No documentation. But are you surprised at this point?

7\. Setting up an environment to build the code was a manual process that
relied on searching through past E-mails for lost tribal knowledge, which
basically amounted to: Try different include and library paths until there are
no errors. Once built I was greeted with 2500+ compiler warnings... (out of
200 or so C++ files).

Oh, and this software ran in airplanes.

~~~
SideburnsOfDoom
>took a development job once, where I learned on the first day: No backups, No
source/revision control, No deployment process, No dependency management, No
bug tracker.

And that is why one should _always_ ask about these important things before
the first day, i.e. during the interviews, before you decide if you want the
job or not. Fortunately I'm in a position that the above would be a near-
automatic "no" from me.

 _edit_ "Near" automatic as there might be a way out if the company knows that
they need to improve drastically and has the will to do it. The other response
"what were you able to do to improve the situation?" is key.

~~~
Philadelphia
Unfortunately, often the answers you get when you ask in an interview bear
very little relation to the reality you find when you start a job.

------
bglusman
Shameless plug: I didn't know about their SonarQube project until I'd built
debt ceiling, but while ruby only at the moment, it is an open source attempt
to provide some mechanics for defining and tracking some quantified
approximation of technical debt. No claims it's perfect of course, but would
love more real world feedback and/or feature requests! Also would like to
extend it to handle JavaScript analysis as well, ideally through a plug-in
architecture that could easily handle arbitrary other languages as well.

[https://github.com/bglusman/debt_ceiling](https://github.com/bglusman/debt_ceiling)

~~~
shoo
Thanks for sharing this. I'm not working on a ruby project, but if I was, I'd
give this a try.

I like how `debt_ceiling` supports `TODO` or `# TECHDEBT` tags.

When I think of technical debt, there are some kinds that don't seem easy to
automatically measure: "we made an assumption 6 months ago that turned out to
be quite wrong, but we haven't had enough time to go back and rework it yet".
This could manifest as poor choices of data structures, or an entire
system/subsystem that turns out to be nonsensical given increased
understanding of requirements. People who understand the problems can manually
tag the code, so it's good to see tool support for these kinds of annotations.

------
brudgers
_There were 39 separate business units represented among the three companies_

The context in which this study was done is quite different from an early
stage startup. This is not surprising since the SEI tends to focus on big
engineering solutions to big problems [which is not to say that its insights
and guidance aren't applicable]. But it is important to realize that they
focus on engineering in situations that are relatively stable over the long
term (29% of systems studied over 10 years old), and almost 1/3 of the systems
studied were embedded where changing the software is particularly difficult
and throwing more hardware at a problem is not cost effective.

------
wellpast
There are some grievances in codebases that do not come with a huge expense.
For example, duplicated code. For another example, overly verbose code. I've
seen a lot of new codebases and these are minor nuisances and are pretty easy
(i.e., take little time) to cope with. I don't classify these as technical
debt because if they cost anything it's always so little.

What costs the most is architecturally unsound code. It's actually not really
the code per se, it's the abstractions and how they fit together--very
specifically, to what extent the abstractions cohere, to what extent they are
coupled, and how narrow are the interfaces. Give me a code base that nails
these qualities (narrow interfaces, minimal dependencies) and all else
(duplicate code, inelegant code) is in comparison NOTHING to deal with.

So the cost of code is almost entirely related to its _architectural
soundness_ (to what degree it's coupled, its interfaces are too wide, etc.)

The biggest and prevelant fallacy of our industry is this notion of technical
debt. Because the notion implies that it's somehow necessary. It's not
necessary.

The ability to build and evolve architecturally sound code is a function of
skillset. If the skillset is there, there is no necessary extra cost. In fact,
for people that have this skillset it costs MORE to depart from instincts and
couple code, create wide interfaces, etc.

When a developer says 'It will take too long to do the right thing,' (i.e.,
build architecturally sound code) he/she is saying, 'I don't have the
skillset.'

I know all the objections here (it takes time to understand your use cases,
etc.), but the objections nearly always miss the fact that architectural
soundness and things like code scope/flexibility are ORTHOGONAL properties.
You can have very specific code that does very specific minimal-viable product
type things ... but this has nothing to do with whether the code is
architectural sound or not.

As long as we continue to think of techincal debt as something that's somehow
necessary and something to be negotiated we will always be missing the point.
The point is that we need to produce workers with the right skillset. There
are so few people that have it--which is entirely a failure in our academies,
educators, training systems... To keep talking about technical debt and not
about how to acquire the architectural skillset is to keep chasing our tails.

~~~
dasil003
> _When a developer says 'It will take too long to do the right thing,' (i.e.,
> build architecturally sound code) he/she is saying, 'I don't have the
> skillset.'_

I'll be charitable and say this may be the case part of the time. But as a
general argument it is bollocks. In the vast majority of production systems
that survive for any amount of time there will be requirements changes, and
sooner or later an engineer will face a choice of shoehorning an attribute in
somewhere it doesn't fit naturally, versus refactoring to more appropriately
model the new business universe. Saying that this won't happen if your skills
are sufficient is a no-true-scotsman argument—it's all tradeoffs, and no
matter how good you are at anticipating change, sooner or later even the best
architect will be blindsided by some out-of-the-blue business requirement.

~~~
wellpast
This is the fallacy I was referring to--architectural soundness has nothing to
do with anticipating change.

The "choice" your engineer faces to shoehorn a feature in [1] versus implement
it in a way that optimizes for soundness (decoupled code w/ narrow interfaces)
is not a choice if the skill set is there--i.e., if he/she has trained and
practiced the right skill set.

[1] (By 'shoehorn' I assume you mean implement it in a way that leaves the
code base in an unsound state.)

~~~
dasil003
You can't just reaffirm that without evidence and an argument. It is self-
evident to me (and I presume every other software engineer here) that
different architectures lend themselves to different types of change. If you
want to make that claim than you probably should write a dissertation about it
because it's quite a controversial claim. But simply stated as if it were
unassailable fact is pushing you right into crackpot territory.

~~~
wellpast
> It is self-evident to me that different architectures lend themselves to
> different types of change.

This is such a vague statement, I have no idea what you mean or how it relates
to our discussion.

My argument is this:

(1) The most costly issue with a code base has to do with its architectural
properties. How architecturally sound it is [1]. Most other issues that we
tend to categorize as technical debt (like code duplication or verbose code,
for examples) pose such tiny cost in comparison to the cost of poorly
architected code [2].

(2) Architectural soundness is an objective property of code independent of
the code's generality, flexibility, etc. Again, it's the degree to which its
components and its interfaces narrow.

(3) There exists a skill set that understands, recognizes and produces
architecturally sound code. [3]

(4) Given an architecturally sound code base and a set of features, a skilled
person can implement the features without violating the architectural
soundness of the code base AND do so without having to pay for it in time-to-
delivery. [4]

I think I'm being fairly clear in my assertions here. They are based on my
long experience. To disagree is to pose a different theory. Do you care to
elaborate on what you mean by 'crackpot territory'?

\---

[1] More specifically, the degree to which its components (modules, classes,
functions) are coupled and their interfaces not as narrow as they could be.
This is an objective property of a code that has clearly delineated components
(i.e., you can quantify dependencies, interface sizes, module sizes.)

[2] Yes, this is a claim based on my 20 years of experience working in myriad
code bases. How to distill and prove this, formalize the argument, etc is
probably work to be done.

[3] This skill set doesn't come cheap and requires _proper_ training, and
currently our industry/academies aren't producing it.

[4] Again, this claim based on my experience and yes there is probably some
good research opportunity here to formalize it.

~~~
dasil003
> _Architectural soundness is an objective property of code independent of the
> code 's generality, flexibility, etc. Again, it's the degree to which its
> components and its interfaces narrow._

I like this definition, and I agree you could quantify this into a useful
measure. However the optimal architecture according to this measure is a
function of the problem domain. You split the modules and define the
interfaces to maximize the encapsulation and minimize the interfaces.

So what happens when the business rules introduces a new cross-cutting
concept? The basis of the architecture soundness is now in question. Do you
define a new module and add n new interfaces? Or do you split the concern into
the existing modules, fattening them in the process, and widening existing
interfaces as well?

~~~
wellpast
> However the optimal architecture according to this measure is a function of
> the problem domain.

I agree with this. But domains typically evolve incrementally -- and using the
skillsets I mention you can evolve your code accordingly. [1]

There is then the question of these cross-cutting, earth-shattering domain
changes you mention. I'm not convinced my argument doesn't have a play here
but regardless if these kinds of major domain changes are happening frequently
enough to matter in this discussion then I think there's a different business-
level problem going on.

It's hard to talk abstractly about this for so long--until of course someone
gets to building out these ideas more formally. Getting into concrete cases
would be more fruitful.

\---

[1] That you _can_ without cost tradeoff is my argument. Today these skills
are so rare so you don't find orgs/codebases that _do_ this properly--and
therefore we have this fallacy that because it's prevalent that it's somehow
necessary. We don't realize it's really a skills problem.

------
pmontra
I remember reading this [http://higherorderlogic.com/2010/07/bad-code-isnt-
technical-...](http://higherorderlogic.com/2010/07/bad-code-isnt-technical-
debt-its-an-unhedged-call-option/)

According to the author call options are a better metaphor for what we call
technical debt but admittedly a more difficult one to understand: anyone who
knows what a call option is raise hand.

------
nbicdugfeiljij
I work in an organisation that has a large technical debt. The organisation
manages at least $A60b of member's assets, and is overseen by the Australian
Tax Office. It also is very visible politically. These three factors make it
extremely conservative in terms of change.

However, change is imposed on it inevitably from the outside, so instead of
making large-scale changes to its architecture, it tries to keep up with the
market by making thousands of small hacks. The original architecture was not
designed to cope with this, so it has become hack upon hack upon hack. I
should be employed (and paid above-average) in this jobs for decades to come.

~~~
zo1
Don't underestimate the value/reward/fun/etc that you can get from a fresh
project. New problems, new development process, the chance to re-iterate past
designs and architectures, integration with existing/old/legacy systems, and a
host of other things.

That's my bit of advice to you on that, as it reads a bit like you're very
comfortable and expecting it to stay that way for "decades to come".

~~~
nbicdugfeiljij
I do work on my own projects that are fun and interesting. But nothing beats a
regular, permanent high-earning job, no matter how boring it is. I personally
know a couple of developers who are stressed financially after working for
"fun" companies.

------
progmal1
I am not sure I see the advantage in taking a metaphor that is a little off
but understandable and replacing it with one that is more accurate, but very
few people understand.

------
alexbecker
I can't make heads or tails of the first figure in the post.

