
We’re Fucked, It’s Over: Coming Back from the Brink - rajbala
http://scott.a16z.com/2014/03/24/were-fd-its-over-coming-back-from-the-brink/?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
======
randall
This is essentially the definition of survivorship bias.

[http://www.fastcodesign.com/1671172/how-a-story-from-
world-w...](http://www.fastcodesign.com/1671172/how-a-story-from-world-war-ii-
shapes-facebook-today)

"In WWII, Allied bombers were key to strategic attacks, yet these lumbering
giants were constantly shot down over enemy territory. The planes needed more
armor, but armor is heavy. So extra plating could only go where the planes
were being shot the most."...

"[A guy in charge] said the military didn’t need to reinforce the spots that
had bullet holes. They needed to reinforce the spots that didn’t have bullet
holes."

Having experienced 2 "WFIO" things in the last 6 months, I agree that
entrepreneurs need to be resilient, but I think it's more valuable to read
post mortems than these "we almost died" posts because they prove more
instructional.

For instance, Aaron's post about Tutorspree is a really useful post for me,
although it definitely has less of a feel good vibe to it.

[http://www.aaronkharris.com/when-seo-fails-single-channel-
de...](http://www.aaronkharris.com/when-seo-fails-single-channel-dependency-
and-the-end-of-tutorspree)

~~~
pseut
> "[A guy in charge]..."

That should read, "One of the most influential mathematical statisticians in
history..."

He also invented the "Wald test".

~~~
novaleaf
and it's actually a really great insight. I can only wish that I would come to
a similar conclusion faced with a related situation.

~~~
pseut
Yeah, that's actually why I think it's important to point out _who_ made the
insight. Every time I've heard the anecdote, it's been presented as some sort
of clever observation that we should all be expected to produce.

But, really, it's a clever observation that was made by a historically great
statistician. And the anecdote leaves out the fact that he had probably been
thinking about this problem before --- he was working as part of the war
effort. So it was probably harder than it looks.

------
JonLim
_> Get all the brains around the table_

An important addition to this: get everyone involved, but also listen to their
ideas.

Having been a part of a sinking ship, the most frustrating thing about having
everyone pour their blood, sweat, and tears into reviving the company is when
the top brass decides to ignore all of the hard work and carry on with their
own ideas that were never cleared with anyone else. They were their own
iceberg.

------
mkrecny
Fresh out of college I was responsible for a WFIO at a 40 person company -
ultimately had to leave.

[http://edu.mkrecny.com/thoughts/how-i-fired-
myself](http://edu.mkrecny.com/thoughts/how-i-fired-myself)

"Um those [backups] got really expensive, so we stopped doing them about a
month ago" sounds painfully familiar.

~~~
paulhauggis
I did something similar about 10 years ago at my first dev job. I forgot a
"WHERE" in an update and destroyed every customer in the live db.

The issue was that my manager never caught it in my migration file and it went
live.

I was lucky that we backed up every day..and didn't really lose a ton of data.

~~~
cousin_it
I never really understood why databases don't have undo.

~~~
specialist
I largely avoid UPDATE and DELETE statements. Changes are new rows, time
ordered, which supersede prior rows. Like an audit log.

Works great. Especially for anything that needs history, e.g. medical data.

~~~
brongondwana
Wrote one of those many years ago, yes - for medical data. It's the right
solution to just about every problem actually, and I'm surprised that it isn't
more commonly taught.

~~~
GFischer
Yeah, logical deletes ftw :) .

One thing I haven't figured out is how to archive old records, I think it's
not a problem for modern DBMSs, but we have a problem at work with 15 years'
worth of historical data.

------
cousin_it
Can't select any text to copy/paste, can't click any of the links in the post
or sidebar. #welcometothenewweb

~~~
gabemart
A transparent portion of the #comments div is covering the body text and
sidebar. If you change #comments position from relative to initial, things
work as expected. I think the problem is that the sidebar is floated and the
float isn't cleared. Setting a clear: both on #comments prevents the div from
overlapping the body and sidebar.

~~~
SeanDav
How does one go about editing the display parameters?

I am a complete neophyte at this, but tried pressing F12 to access various
debugging tools. I could not find any reference to the comments.

Also interested how you discovered this in the first place?

~~~
gabemart
In chrome, I right clicked on the body text and hit "inspect element". That
brings up element inspector with the relevant element highlighted. The
highlighted div had the id "comments". Hovering over that element in the
inspector pane shows you the dimensions of the div in blue and the padding in
green in the main windows. This makes it pretty easy to see that the comments
div is covering everything else.

The css for #comments in the inspector pane doesn't specify dimensions for the
div, so it should be the size of its child elements. None of the child
elements overlap the body text or sidebar, so something else is causing the
parent div to cover those elements.

The div directly before #comments is the sidebar, and clicking on it in the
inspector pane shows the css floating it.

I then just added a clear: both to #comments using the inspector and it fixed
the problem in the main window. You can add css to an element in the inspector
in chrome by hitting the "+" in the right-hand pane (there are other ways, but
that's probably the easiest).

------
eterpstra
"Nothing is beneath a leader in times of crisis"

I've worked for people that believe this, and others that don't. Nothing
builds morale and loyalty like rolling up your sleeves and doing the hard
stuff with your employees.

I once had a project manager who couldn't write a lick of code, but whenever a
deadline approached or something blew up, he'd be right there at 10pm with
everyone else bringing in fresh pizza, testing whatever he could, and giving
pep talks. Contrast this with someone that says, "get this done by tomorrow,
OK?" and then walks out the door. Who do you think is going to have a higher
performing team in the long run?

~~~
incision
_> "Who do you think is going to have a higher performing team in the long
run?"_

The project manager who manages the team, schedule and risks such that things
don't routinely blow and run down to the wire would be preferable to either.

~~~
eterpstra
lol. In a perfect world where bugs never happen, estimates are always perfect,
and last-minute top-down requirements changes never occur, I would totally
agree with you.

------
malanj
I had serious WFIO case a few years ago. We were building a crypto solution on
J2ME phones.

The solution was days away from roll-out with our first big corporate client.
Late on a Friday afternoon we were busy with final field testing - paying
students a few $ to use their phones to test the app.

As the test data came in we realised we had a major problem: a small % of the
phones weren't returning the correct test vectors for hashing algo.

After checking for obvious user error we came to the conclusion that something
big was broken. Specific firmware sets didn't execute the crypto part of the
code correctly. The entire value-proposition was that it works on every phone
that can run an app, so it was a pretty big deal. I thought we were totally
fucked.

We didn't sleep for two days and finally found the bug in the way the phones
implement a bit-shift operation (doesn't carry a bit about 1/10000000 times).
Then had to figure out a workaround that was still fast enough.

We shipped a fixed version before the end of the weekend, but I wouldn't wish
that kind of stress on anyone.

~~~
saryant
I ran into something similar a few months ago. There's a bug in Oracle JDK
7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that
version's Diffie-Hellman cypher suite.

Tracking that bug down was a fun three days.

------
resu_nimda
I don't understand the meaning of "crack the egg with a sledgehammer." I would
think it means "to use way more force than is necessary," but the context is
more along the lines of "get the problem solved at all costs."

I also followed the link to the other article where that phrase is used[1],
and found something a bit concerning: "Nawaf moved the entire engineering team
over to work on it. He called them all in to work nights and weekends until it
was fixed, [...] Nawaf saved our bacon."

Umm, what about those engineers? I'll give the benefit of the doubt that they
were rewarded appropriately, but the wording here seems to almost deliberately
stoke the developers vs. management flames, especially coming from a VC...

[1] [http://scott.a16z.com/2014/02/03/harvey-keitel-
ceo/](http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/)

------
TheCraiggers
Am I the only one that thought the query string was actually more interesting
than the article?

?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

I'm glad that ycombinator isn't listed as the source, at least. I think it
would be fairly interesting to see what kind of data they have about various
social medias and link propagation.

And also, a slight sickening sensation that marketers are doing this. Not that
I'm surprised, it's just that I'm sure they have papers written saying that
"Twitter has a 64% link click rate if you write your post like X, whereas
Facebook can achieve as high as 70% if you do Y."

I know it has been going on for years on the web, and decades for other
marketing, but here it's plain to see. It fills me with unease, like I'm no
longer in control and what I like in life is already planned out.

~~~
michaelbuckbee
This is a pretty standard Google Analytics Tracking URL [1] - setting aside
the Googleness of it (which raises a different set of issues) I look at it
like the modern equivalent of an Apache Access Log.

Also, not sure if this will make you feel better, but most places that are
doing this type of in-depth analytics, etc. are much more often trying to bend
themselves to your will ("what feature, benefit or product that people will
get value out of") than to trick you into liking something you otherwise
wouldn't.

One way of interpreting that query string is as the answer to an implicit
survey question of "How would you like to get the stories we write?" the
answer in this case being: "Twitter" [2].

1 -
[https://support.google.com/analytics/answer/1033867?hl=en](https://support.google.com/analytics/answer/1033867?hl=en)
2 - Obviously in this case it's pretty well statistically messed up by that
link being posted to HN, but you get the point

~~~
eli
I think the "Googleness" is in your head. Those parameters were defined long
before Google bought Urchin.

~~~
michaelbuckbee
I was referring more to the differences in Google vs some other solution
tracking your behavior on websites.

------
pgrote
"It's never really as bad as it seems."

Very true. Many times when something happens the automatic response is to
focus on the worst case scenario instead of the immediate problem at hand.

Also, I couldn't select any text to copy and paste.

~~~
not_paul_graham
This is great advice even for students because sometimes the things that
younger people (anecdotal: me) give a lot of importance to and feel terrible
for screwing up end up not mattering at all.

1\. Things like validation from peers / popular kids

2\. Screwing up on that major test that sometimes "defines" where you end up.
A lot of suicides result from academic pressure and failure (plus the
environment/parents hinting that you can't succeed in life if you can't do X,
Y or Z properly).

3\. The academic thing is worse if you are artistically inclined and have
science and math thrown down your throat telling you that humanities are for
those that can't cut it in science (I know friends that have had a variation
of this conversation with their kin/parents, etc) quite true in countries like
India / Pakistan, Korea and perhaps even China.

4\. Humiliation/shame from something that seems drastic in the moment (Getting
rejected by the cute girl in front of all your friends and classmates. Yea
that sucks). Getting labeled sucks too, a lot of girls can be mean to their
own gender at that age which is absolutely disgusting.

Unfortunately for most people it is something that they only figure out in
hindsight.

~~~
Fomite
I had an Economics professor who gave a small lecture to our class after we
got our midterm grades back.

"This midterm is one third of your grade. This class is one of four you are
taking this semester. This semester is one of eight you will take at
$University. And while people will tell you otherwise, your time in college is
a small part of your life. It will be okay."

------
orthishappended
The most interesting take away from that was how resilient companies appear to
be despite their leaders best efforts to sabotage everything. I think the real
lesson is (and I don't like it): if you want your company to survive you need
to be good at spin.

In the case of hotmail, how did he ever let a situation arise where one bug in
a nightly cron script could obliterate his users emails?! No backups because
they cost too much?? "the sun, the moon and the stars lined up against us for
it to happen" No, you were just reckless and too busy chasing growth.

With Ironport, he says his whole business was built on the back of anti-spam
yet they were totally unprepared when their partner who provided the anti-spam
tech they used pulled the rug from under them (despite knowing full well this
day would come). The real lesson here should be: try not to outsource the key
component upon which your entire business depends.

I had to laugh at the end when he says the only time he's had a failure was
when he was using his own limited resources to fund a business and not
investor money. Well yes... mistakes are easier to hide/absorb when you're
swimming in cash. Not so much when things are tight and you can't afford to
learn after the fuck up.

------
JoeAltmaier
Some survivor bias there, but an inspiring message: always continue as if
you're going to survive. All the real survivors did that.

------
mfrommil
Both Hotmail & Ironport reacted with a customer-centric focus (e.g. Hotmail:
"How do we recover their emails? What else can we provide customers in the
future? What steps will we take so this never happens again?")

I would guess a lot of companies in a WFIO situation that take a CYA approach
rather than customer-first face a worse fate.

------
zxcvvcxz
Question - why did IronPort cancel the contract with their customers if they
just got the feature implemented that they seemed to want? Was it a done deal
- i.e. they were losing those customers anyways?

Also, survivorship bias much? Two examples --> conclusion? Was nice hearing
about those company's histories though, didn't know about the Hotmail thing.

~~~
chrisweekly
"Instead of Symantec canceling the contract, we went on the offensive and
faxed a letter to all of our customers cancelling the contract with THEM
[Ed.:SYMANTEC]– a position of strength."

I initially misread that too. He's actually saying they faxed a notice to
their customers that they were canceling the contract with Symantec. It's
worded poorly.

------
sswaner
The same principles were true and applicable in a much larger near-implosion
that I experienced from 2008-2012 working for a "too big to fail" financial
company. Strong, out-front leadership, a strong team led the way. It took time
for us to realize that it wasn't as bad as it seemed. We had to learn that the
external pressure from the media, public and even friends and family was only
a distraction.

In this situation it was quickly apparent that my role was to focus on a
solution, not dwell on the problem or the cause. This should be true for
everyone on the team, unless you are the specific individual who caused a
global financial meltdown, deleted all the email or caused a critical failure.

~~~
tsunamifury
Actually that was a problem due to the culture of stupidity and willful
ignorance of risk, not some keyboard mash that caused the markets to crash.

It might be one of the rare situations where it was so bad, that after
stabilizing the situation, many of the responsible parties needed to be put
down.

I know from my experience working with Wachovia, the gross incompetence,
system wide narrow minded greed and the fuck-everyone-but-me attitude was so
bad it was mind boggling.

It was not a mistake, this was a concequence of the culture and goals of those
companies.

------
Susan_we
I really admire Scott from what he's written. He can be an inch close and
still get out of WFIO situation. And he seems to know the difference when it's
doomed like with e-commerce business. So keep rowing, Scott!

------
Mz
I am disappointed to see the top comment here criticizes this as "survivorship
bias." Maybe the survivors have something useful to communicate that can help
others survive what looks "impossible" when you first run into it? Maybe
that's the point?

Though this probably shouldn't be exactly a surprise to me given how much shit
people give me any time I try to talk about getting well after doctor's
basically wrote me off for dead in some sense. No one wants to learn from that
either and I honest to god don't get it.

------
mathattack
Great points, especially leading from the front when times are tough. I didn't
think Hotmail would have survived the outages, but they came out just fine.

------
espressopowered
Nice article.

I particularly agree with the point on leadership. If you can motivate those
around you, people perform so much better and can sometimes come up with
better solutions. Doesn't always guarantee you coming back from the brink, but
stands you in good stead.

------
lauradhamilton
"We're Fucked, It's Over" strikes me as a pretty dramatic response to a
technology vendor failing to renew a contract. Unless they have extra-special
access to customers or customer data, whatever they do can be replicated in-
house.

------
ajsoh
Fascinating to read a 1st person account of a near-death moment. "Companies
are damn resilient" \- I think entrepreneurs are damn resilient, too, for
making it through crises like this over & over...

------
ssully
My palms got sweaty just reading the hotmail portion. I think someone could
make a really great horror book that is just a collection of stories like
this. I know it would give me nightmares.

------
loladesoto
if it's never as bad as it seems: does this make a case for optimism or
realism on a team?

regardless, that's precisely why you need a cofounder who balances you.
whether it's a different perspective, questioning some of your choices, or
holding you accountable (and holding you up): you need it. you'll make an
order of magnitude more poor choices on your own.

------
caycep
Hm, WFIO is kind of like "welping a fleet" in EVE...

------
dsugarman
how did you build the spam filter? bayesian classifier?

~~~
herdrick
No, not in 1997.

------
michaelochurch
_The leader needs to be the first one there, the last one to leave, and be
willing to do anything it takes – like answer customer care calls or
personally drive a replacement part to an irate customer. Nothing is beneath a
leader in times of crisis._

I like this, because it says two things at once, both true.

First, if you're in a position of ownership/authority but you're not willing
or able to do the grunt work, you won't have the credibility for long. People
will follow orders out of fear of getting fired, but you'll never get more
than the bare minimum. There's a point where leaders are replaced or
outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites)
and after that point, the organization can't even motivate shit to stink.

Second, it might be that _it 's just not worth it_ to lead from the front.
That means that you're not _really_ in a position of leadership. You might be
a middle manager who realizes that the people above you will never buy in to
what you're doing. Then you'll probably lose that desire to make those
sacrifices. That's fine. You shouldn't tie yourself to the mast, at that
point, because you're not really in a leadership role anyway.

