

Technical Debt - baha_man
http://martinfowler.com/bliki/TechnicalDebt.html

======
mattjaynes
Great article. It reminds me of another dilemma I came across earlier this
week.

I had a 280MB xml file containing about 120,000 records. I needed a way to
parse out a subset of the records (about 15,000) and then put their data into
a db.

I was developing on a vps with only 256MB of RAM, so I wanted to avoid memory
intensive operations. I'm using ruby, so I started looking into how to
'stream' the xml with ruby in such a way that I could read if it was the type
of record I wanted to keep, somehow manipulate and store the data and move to
the next record. The more I looked into that strategy, the more complex and
ominous it seemed. I just really wanted to find a simpler way that wouldn't
require all the apparent tediousness of streaming the xml file (it may seem
simple enough, but there are dependencies you have to install, api's you have
to read and learn, etc). There were just too many moving pieces and it made me
feel eery about the outcome. Another aspect of it was just pure laziness. I
just don't care how I get that data into the db - I just want to get it in
there.

Fortunately, the glorious wonderfulness of linux utilities saved me from a
long tedious solution. Behold:

    
    
      csplit -q catalog.xml '/<title_index_item>/' '{*}'
    

This splits the large xml file into sub-xml files that start with 'xx' by
default, followed by an incremental number, xx10004 for example. It splits the
file based on the <title_index_item> tag - which is the tag for the items I
want. See `man csplit` for more info...

    
    
      find xx* | xargs grep -L 'label="instant"' | xargs rm -f
    

The 'find' lists all the sub-xml files, then we grep for the filenames that
_do not_ contain what I'm looking for, then delete those files. I'm left with
a directory of 15,000+ xml files with just the type of data I'm interested in.

    
    
      find . | wc -l
    

This command is just so I can track the progress of the operations. Obviously
`ls` would return too much data, so we pipe the file list to `wc -l` which
gives us the number of lines - which in this case is the number of files in
the dir.

So, two lines of code on the command-line instead of a far-more complex
ruby/streaming-xml solution. Now I can have a ruby script process each
individual file and add the data to the db - a much simpler problem to deal
with.

My point is that you can accumulate code debt by doing the seemingly 'right'
solution sometimes. There are probably coders that will cringe that I just
used a couple of shell commands to do this rather than write up a long, well-
documented, properly OOP, TDD, etc "right" solution. However, the "right"
solution in that case would have accumulated code debt - more code to
maintain, more moving pieces, more things that can go wrong. I'd trade
maintaining two lines of shell code over 10's or 100's of lines of ruby code
and it's dependencies any day of the week.

~~~
gaius
In similar situations I just use a SAX parser, that's what it's for. I don't
know about Ruby, but it's quick and easy in Python. And once you've written
one SAX application, you can easily use it as the basis for the next.

~~~
mattjaynes
Totally. If I had had to do anything much more complex, I may have had to
resort to SAX. I was just too lazy to mess with it for this - and I'm a bit
OCD sometimes about writing the least amount of code possible ;)

------
gruseom
It's good to see Ward Cunningham recognized for one of his many great
contributions. I once heard a whole NPR piece about wikis and their history
which didn't mention Ward as their inventor. It only occurred to me later that
I should have written in.

The video of him talking about the debt metaphor is worth watching, especially
the last minute where it gets subtle.

------
akeefer
I've thought a lot about why technical debt seems to be such a hard concept
for non-engineers to grasp; certainly given how few business people or
managers seem to understand it, it clearly must be difficult. I think it's
because A) it only affects a truly long-lived, stateful process, like a
software (or many other types of engineering process) where the output of one
phase is the input to the next phase and B) there just aren't that many sorts
of processes that occur in ordinary life.

Sure, in any field you can screw up in such a way that it hurts you in the
future by damaging your reputation or annoying your customers, or make bad
technical investments that turn out to cost you. But in engineering
disciplines, doing your core activity (engineering) poorly can hurt your
ability to keep doing your core activity (engineering) in the future.

If you're in sales and you cut corners with one client, it doesn't make your
phone calls take longer. If you're in a restaurant and you cook things too
fast or cut corners on ingredients, it doesn't make your future cooking take
longer. If you're a doctor and you're too hasty and mis-diagnose a patient, it
doesn't make your future surgeries and checkups take longer. But in software,
if you're too hasty adding feature A, it _does_ make your future features take
longer to develop. And that's because in software, your future features are
built on the previous ones, whereas with sales, cooking, or medicine the
duration of any given prospect/customer/patient is relatively short-lived and
finite.

That's my best theory as to why it's so hard for non-engineers to understand.

------
10ren
Technical debt is appropriate in prototype code, where you aim at getting
something you can play with as a user; and also get a feel for its technical
feasibility. This works fine.

Unfortunately, I ended up wanting to go back to the code to reuse/build on my
approaches to the key technical problems... but the hacked mess was tortuous
to read or understand (interest payments). So the dilemma is: how to write
readable code, when it's only for a prototype and so not worth the effort?

My current tradeoff is to make the prototype code _too_ simple, by cutting out
tricky special cases, and by deleting approaches I coded and all my comments
discussing them, assessing issues etc. That is, make the code "workmanlike".
It might be stupid, incomplete, inefficient and incorrect - but in a _crystal
clear_ way.

I actually find this discipline incredibly difficult, and I often lose days at
a time exploring alternative coding. This often pays off; but the biggest
payoff comes from having a clear and concrete basis to build on. I've just
finished the current prototype, and many solutions to technical issues have
now clicked into place as obvious to me - because I can now _see_ them. So
that works well.

And I also have something - incomplete, buggy, inefficient etc - that I can
play with as a user. Ugh. In other words: _a crystal clear sense of what needs
to be improved._ Maybe..."auditable"?

------
kailashbadu
In a nutshell, could it be argued that kludges are fine as long as the
programmer?

• knows the exact location of the hacks.

• Is aware of the repercussions and is assured that the system can get away
with the kludges for the time being without any hindrance in the system’s
effectiveness to get things done.

• Knows how to fix them if need be.

• Communicate the situation with others, including the fellow programmers and
the managers.

------
DannoHung
Can I securitize technical debt?

~~~
run4yourlives
If you are a contractor, it can actually be profit! :-)

------
joe_the_user
A great, crucial article.

Even after reading the article, I think that folks often don't understand that
Technical Debt is _not_ a "bad thing" but a thing that must be understood and
managed.

When you understand this, it puts software engineering into a very clear
perspective.

~~~
jsdalton
> a thing that must be understood and managed.

That's true, but _how_ do you manage it? Especially when it's so difficult to
measure in the first place.

One idea that popped into my head would be to keep a running tab of technical
debt that's being accumulated in a project. It's tough though, because
developers who are in "accumulating debt" mode tend to be the ones with the
least amount of time to spend on upkeeping a document that tracks their debt.

~~~
gojomo
I leave a lot of TODO and FIXME comments in code (which are then tracked and
highlighted by my IDE). These range from code that may actually be broken on
certain rare or unexpected corner cases (though not dangerously so), to known
inefficiencies, to areas where the rough idea for a much better approach
exists but can't yet be fully developed.

Very roughly, they are a measure of known technical debt.

~~~
rs
Good practice. Do you end up cleaning them up ?

~~~
gojomo
I think I wind up adding new ones at about the same rate as old ones
disappear, but I'd have to run a report over the history to know for sure.

It's always nice when I come across a note that's been made irrelevant by
other parallel improvements along the same theme -- even though the note
wasn't the impetus for the other changes. It's like finding a $20 on the
street!

------
biohacker42
I'm not sure debt is a good metaphor for code complexity.

Debt grows logarithmically, if the complexity of your code base is growing
logarithmically you're doing great.

The problem is software complexity grows exponentially.

And that's why you can't manage it like you can manage high interest debt.

You can try to manage it, but I have yet to see that management succeed.

Every single time a company has tried to manage technical debt, the debt
simply goes unpaid.

How many people here who've worked on a sizable and mature code base can name
_temp_ files and variables that have turned absolutely permanent over the
years?

That's why software complexity, in large organizations, should be approached
like an infestation. Erase and eradicate on sight, don't try to save some for
later.

------
alrex021
First time I came across this metaphor is not to long ago reading the "Getting
Real" book by 37signals. <http://gettingreal.37signals.com/>

------
skmurphy
Gerald Weinberg refers to this same concept as "maintenance debt" in his four
volume "Quality Software Engineering Series" from the early-mid-90's. I think
it's an independent formulation of the same concept. The volume most concerned
with it is Vol4 "Anticipating Change" [http://www.amazon.com/Quality-Software-
Management-Anticipati...](http://www.amazon.com/Quality-Software-Management-
Anticipating-Change/dp/0932633323)

The whole series is definitely worth reading.

------
jwb119
the article is good, but i can't help the feeling that it is just applying a
clever name to a common sense principle that is relevant to just about every
field (including finance and technology)

~~~
StrawberryFrog
If giving it a clever name is the only way to explain it to management (which
may be the case here) then it's worth it.

~~~
Clark76
"Technical Debt" is a surprisingly self-explanatory term. I have used it to
great success with my management team when arguing for time for refactoring,
code reviews, and other non-user facing work.

~~~
StrawberryFrog
Exactly. technical debt is a great tool for explaining to non-technical
management why you'd want to invent time in refactoring, reviews etc that have
no obvious tangible result in the finished product (short version: it's a drag
factor makes it harder to add features and keep the bug count down later).

It's also a good tool to allow the management to do what management does best:
take the technical information and make management/financial decisions based
on it, on how much investment can be made _now_.

It turns a binary either/or choice: devs saying "Our code is bad, we need to
rewrite" into a quantitative one: "How much technical debt do we have? Is it
increasing? Can we reduce it?".

------
alecco
Technical Debt is a very good new metaphor. This article looks like a quick
blogspam/linkbait/younameit.

This guy writes books on UML.

------
djahng
That's just like Herbert Kroemer's Conservation of Misery. PhD students in ECE
at UCSB know all about that.

