

Five whys: The startup immune system, Part 1 - wheels
http://venturehacks.com/articles/five-whys

======
etal
A few years ago I worked for a smallish company where the repeated "why?"
would consistently release untold nightmares. The turnover rate was high, but
there were a few old-timers who would tolerate a historical/political inquiry,
and a few more whose reactions revealed things about history and politics in a
different sort of way. At a glance, the product designs and manufacturing
procedures appeared to have been born from pure ether. (This company made
medical devices.) Design decisions were undocumented; what documentation did
exist was _withheld_ out of general mistrust.

Problem: Applying the drug to the latest version of the device in the usual
way gives a messy coating.

\- _Why is the coating messy on this design?_ Because the device's surface
can't hold this much drug.

\- _Why are we applying so much drug?_ Because the previous products used the
same dosage.

\- _Why was that dosage chosen?_ Because, in an animal study using sample size
N=1-3 for each of three different dosages, this did slightly better than the
other two. No further investigation was ever done. EOF

\- _Why can't this device cleanly carry as much drug as previous designs?_
Because the surface properties and geometries are different.

\- _Why did we change the design? What properties did the other design have
that resulted in a clean coating?_ Because a partner company wanted out of the
biz, created a durability test no existing product could pass, and used that
to torpedo our product. And, wait, that design had a problem with the coating,
too.

\- _Why didn't we catch these coating problems in previous products?_ Because
quality is fucked in this pilot facility, and our other manufacturing facility
is still trying to get up to speed with the first product made with this
manufacturing method, and unable to duplicate our allegedly flawless results
from two years ago.

\- _Why are we unable/unwilling to do serious QC work here? Or even basic
science?_ We think the CEO doesn't intend to pass regulatory requirements or
even launch most of these newer products, so it doesn't matter. These projects
are window-dressing for investors. Yes, _of course_ we're all looking for new
jobs on the side.

As it turned out, the real benefit to asking all of these whys was that I was
able to abandon ship at just the right time. Shortly before I left for another
job, having challenged the CTO about the scientific feasibility of his latest
escapade during a meeting, a senior engineer showed me some years-old pictures
of a device with exactly the same physical design as our flagship product.
This young engineer was enlightened.

------
run4yourlives
This is great advice and can be used everywhere.

Beware that if you work for BigCorp this will make you unpopular.

~~~
pmjordan
The statement "Because that's the way it is" springs to mind.

~~~
aaronblohowiak
When I worked for a larger company, I had the rare pleasure of working with
someone who was at once brilliant and patient (something he actively
cultivated.) When trying to get to the bottom of things, I usually was granted
a purview into the historical technical and political _history_. While this
was illuminating for a while, eventually it boiled down to: lack of approval
to pay off technical debt -- the compromise was made knowingly and was the
best decision at the time. Of course, this was something I thought to be true,
but only really learned it after coming to the same conclusion time and time
again.

~~~
azanar
I've had the benefit of similar illumination where I'm employed, although
without the same level of patience. The same explanation was given time and
again there, too; incurred technical debt due to prudence. They were small,
and thus didn't have the financial wherewithal to afford a long term outage
should some of that debt sudden become subject to a margin call, and so I dug
deeper. And from that the true reason came out of why things had gone off the
rails so bad. The political history of the company had been one where ego and
authority ruled. The technical debt would not be repaid, because it would be
an admission that the incurring of such debt was not _inevitable_. It would be
a sign that one or more wrong decisions was made. And, due to the history of
authority rule, no one would dare question the decisions, assuming both that
the management was approaching the situation with full information and
rationality, and any one else should assume to have neither. From what I've
gathered, I'm not so alone in the universe with these experiences. But when
the term "political history" comes out to explain something, I'll generally
start reaching for my fine tooth comb, to better understand who it matters to
and why.

~~~
curiousgeorge
What exactly is meant by incurred technical debt? How do you have "margin
calls" on technical debt? Totally confused.

~~~
jerf
I agree with pmjordan's answer, but thought I'd add an example: Your company's
product log messages of some kind into a database. The developers were lazy,
and put everything in one table with the classic "key, value" SQL antipattern.
Subsequently, it was discovered that values weren't enough, you sometimes
needed arrays of values (or hashes of values, or ...), so a (probably crappy)
serialization format was written for the values so they could be arrays.

Suppose now you were hired in to this position and you actually do a lot about
databases. Now, your company gets a call one day from your biggest customer
that while they've been putting up with your product getting slower and slower
over time (without telling you of course), it's finally gotten to the point
where "queries take over an hour and we really need them in less than 5
minutes"/"our daily reporting process now requires more than a day to
complete" (which means the system clogs up with nothing but reporting
processes, eventually)/"we've been sued and the court has mandated that we
turn over some data by next week and we compute it'll take two months"/"the
CEO just tried a query that took an hour and now he wants to cancel the
contract"... or so on, you get the idea.

This critical problem comes back down to you, and you realize that the only
way to fix it is to entirely restructure the databases using better practices.
Conceptually, this is trivial. However, since _every single part_ of the
system is built around this data system (and of course, there's no
abstraction, everything uses the key/value store directly), the combination of
the amount of time it would take to write the new schema, modify every program
file in source control to use this new schema, write the migration script, and
potentially even the time to run the migration script for the customer (during
which time they will be off-line, which could be days, and which could be
anything from unacceptable to impossible), oh, and testing, don't forget
testing, could in fact be longer than your company can survive with a pissed-
off major customer.

And if one customer is pissy now, it means every single other customer you
have is a ticking time bomb.

(Everything in this message comes from things I've seen more than once in my
professional career, except the hypothetical about getting sued (thank
goodness). I'm not providing details because there are thousands of companies
that could fill in the above!)

Your technical debt margin just got called. You couldn't cover it. It wasn't
even your technical debt, but who cares? Buh-bye!

~~~
run4yourlives
You just described my current day job. lol.

------
ScottWhigham
Looking forward to more.

