"In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."...
"[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."
Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional.
For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it.
May i say how wonderful it is too see a relevant Hark! rather than a relevant XKCD for a change. Class, tonight's homework is to find a relevant Dinosaur Comics. For extra credit, a relevant Achewood.
Perhaps you're overanalyzing this post? For one, he doesn't appear to be postulating some grandiose startup theory. I read the post as "here is some bad stuff that happened to me and here's some advice if it happens to you."
Furthermore, have you considered that a general partner at Andreessen Horowitz has also seen plenty of what causes companies to fail? Maybe he just didn't outline it here?
Not to mention his last paragraph: "Sometimes it really is over, and I believe that one of the clearest signs is when you are completely out of cash. But right up until that point, when the music is still playing, when your team is still driving hard, you have a shot – and usually a decent one."
This is a great paper, but I really wonder how much difference it made.
The conclusion says
The greatest probability of being destroyed is .534, and occurs
when a plane is hit by a 20-mm cannon shell in the engine area.
The next most vulnerable event is a hit by a 7.9-mm machine gun
bullet on the cockpit.
I think most people would suggest that these are the parts worth protecting the most.
Another thing, I'm not sure that the method is unobvious. This is not the fault of the author but of the fairly breathless retellings that sound like an episode of 'Numb3rs'
Yeah, that's actually why I think it's important to point out who made the insight. Every time I've heard the anecdote, it's been presented as some sort of clever observation that we should all be expected to produce.
But, really, it's a clever observation that was made by a historically great statistician. And the anecdote leaves out the fact that he had probably been thinking about this problem before --- he was working as part of the war effort. So it was probably harder than it looks.
"than these "we almost died" posts because they prove more instructional"
Agree.
Also noting that hotmail in particular (1/2 of the n=2) was a free service.
There wasn't much that the customers who lost their mail could do legally (most likely) because they weren't paying for the service. So I am guessing it wouldn't be something that you could hire a lawyer to pursue [1] or even band together for a class action.
[1] Which is a really important concept actually. If you are providing a service to a customer and charge either nothing, or a very cheap price, and have a good contract, the chance of someone hiring a lawyer to pursue an action is pretty small.
On a slightly related note, i used to ask at interviews, "What was the worst mistake you've ever made?".
I found out later when stopped by the (UK) police once, that their "Have you ever been in trouble with the police before?" is a similar one. After i answered it pretty dismissively, he said "It's amazing what some people will tell you".
Upvote to the top replier for this anecdote, which i'll keep for special occasions:
> "[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."
An important addition to this: get everyone involved, but also listen to their ideas.
Having been a part of a sinking ship, the most frustrating thing about having everyone pour their blood, sweat, and tears into reviving the company is when the top brass decides to ignore all of the hard work and carry on with their own ideas that were never cleared with anyone else. They were their own iceberg.
I remember reading a while ago and cringing at this so bad, this needs to be a cautionary tale of version control, development/staging/production environments and ... backups.
Thanks, but ultimately, leaving when I did was extremely positive. I ended up in NYC in the employ of Techstars which turned out to be a much better place for me.
FWIW, whoever built a system that had no redundancy or disaster recovery plan or even properly separated environments was responsible. You were just unlucky.
You may have been the straw that broke the camel's back, but you shouldn't feel bad about that. I recently had a junior sysadmin learn the hard way about BEGIN TRAN/ROLLBACK/COMMIT. But that's all it was, an embarrassing lesson as he explained to a customer why they'd need to wait a little bit longer.
Even a major "DELETE FROM Foo" should just immediately get the DBA switching to read-only mode, and then restoring using the transaction logs. It's annoying, it causes a bit of downtime, but it's not a "Big Deal".
Yeah, that would be a nasty mess for us too... the change would replicate to all our databases and we'd have to stop the world, restore from the previous day's backups, re-run all the database binlogs up until the point where the disaster happened. It would be a couple of hours of total downtime - not a happy place.
But we do have daily backups and binlogs in multiple places, so it wouldn't actually lead to data loss. It really sounds like you were lucky NOT to be there!
For some reason this is one of the worst of these I've ever read.
I've personally f'd up pretty bad restoring a database to the wrong the site and blowing away 3 months of content. We didn't ever do any offsite backups and I just lucky to have an in with our hosting company and they managed to get everything back but those 3 months.
Once bitten, you're always way more cautious but it's easy to get lazy again from time to time.
Great story, I remember reading it the first time around. How they didn't explode sooner from that set up and and managed to get as far as they did was probably the most shocking part of it.
Wrote one of those many years ago, yes - for medical data. It's the right solution to just about every problem actually, and I'm surprised that it isn't more commonly taught.
One thing I haven't figured out is how to archive old records, I think it's not a problem for modern DBMSs, but we have a problem at work with 15 years' worth of historical data.
A transparent portion of the #comments div is covering the body text and sidebar. If you change #comments position from relative to initial, things work as expected. I think the problem is that the sidebar is floated and the float isn't cleared. Setting a clear: both on #comments prevents the div from overlapping the body and sidebar.
In chrome, I right clicked on the body text and hit "inspect element". That brings up element inspector with the relevant element highlighted. The highlighted div had the id "comments". Hovering over that element in the inspector pane shows you the dimensions of the div in blue and the padding in green in the main windows. This makes it pretty easy to see that the comments div is covering everything else.
The css for #comments in the inspector pane doesn't specify dimensions for the div, so it should be the size of its child elements. None of the child elements overlap the body text or sidebar, so something else is causing the parent div to cover those elements.
The div directly before #comments is the sidebar, and clicking on it in the inspector pane shows the css floating it.
I then just added a clear: both to #comments using the inspector and it fixed the problem in the main window. You can add css to an element in the inspector in chrome by hitting the "+" in the right-hand pane (there are other ways, but that's probably the easiest).
The easiest way (in Chrome, but other browsers will have something similar) is to right click pretty much anywhere on the text and click "Inspect Element". The developer tools will come up with that element in the DOM highlighted (it is a div element with id "comments"). The easiest thing to do is then to right-click on that element and select "Delete Node".
By the way, when the "what can we do now that we couldn't do in the 90s" debate comes up it always occurs to me that the majority of the applications I use are far more malleable due to being web applications built on its fundamentally open (if admittedly complex and fragile) tool set. So instead of thinking "well, this UI sucks", I can actually fix it myself if I care enough to.
You see that the text is not highlightable. In Chrome, for example, you might right click that text and select "Inspect element". You can see that while you clicked on a paragraph of copy, the element that gets inspected is the #comments div, which should live below content of the article. Under the "Styles" tab of the right pane of the developer tools, you can modify existing css rules or enter new ones under element.style or #comments if you like. Adding a "clear: both" rule fixes the issue.
(In Chrome) If you right-click anywhere on the text and select "inspect element" it will select the #comments div because it covers the whole page. Then on the right-hand side of the dev tools you can add to "element.style". Adding clear:both; to this fixes the issue.
Yeah, actually, I don't read articles anymore that do shit like that. Also: If I end up on a website that has some sort of popup (like us on FB, sign up here), I instant-close it. I don't care how good the content is. If they annoy me, I won't read it.
If enough people do it, reading time will go down and bounce rate will go up. If that happens, someone might notice and decide to change something. I think there's no other way than that to tell them that their approach sucks.
Just a layout bug caused by improper float clearing, causing the #comments div to obscure the content and the sidebar. OP, if you're reading this: Just add your "clear" class to that div, and you're golden.
I had this problem also but I was logged in to wordpress.com (or .org, I don't even know, but it had the black bar at the top from WordPress) and when I signed out, that problem went away immediately.
select all still works (ctrl a), but yeah, horrid that they are doing it with css instead of at least having the decency to do it with javascript so I can easily disable that.
I've worked for people that believe this, and others that don't. Nothing builds morale and loyalty like rolling up your sleeves and doing the hard stuff with your employees.
I once had a project manager who couldn't write a lick of code, but whenever a deadline approached or something blew up, he'd be right there at 10pm with everyone else bringing in fresh pizza, testing whatever he could, and giving pep talks. Contrast this with someone that says, "get this done by tomorrow, OK?" and then walks out the door. Who do you think is going to have a higher performing team in the long run?
I'd rather be working for the project manager that tells everybody to go home at a reasonable hour, get a good nights sleep, and come back fresh in the morning. That's the most productive. It's amazing how often an intractable bug in the evening becomes an, "ah hah, it's x" in the morning.
>"Who do you think is going to have a higher performing team in the long run?"
The project manager who manages the team, schedule and risks such that things don't routinely blow and run down to the wire would be preferable to either.
lol. In a perfect world where bugs never happen, estimates are always perfect, and last-minute top-down requirements changes never occur, I would totally agree with you.
In the truly long run, one of those teams will either burn out or die of a collective heart attack from all the late nights eating pizza at its desk. Still, it's nice when someone chips in to do additional testing before release.
I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones.
The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app.
As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo.
After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked.
We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough.
We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
I ran into something similar a few months ago. There's a bug in Oracle JDK 7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that version's Diffie-Hellman cypher suite.
I don't understand the meaning of "crack the egg with a sledgehammer." I would think it means "to use way more force than is necessary," but the context is more along the lines of "get the problem solved at all costs."
I also followed the link to the other article where that phrase is used[1], and found something a bit concerning: "Nawaf moved the entire engineering team over to work on it. He called them all in to work nights and weekends until it was fixed, [...] Nawaf saved our bacon."
Umm, what about those engineers? I'll give the benefit of the doubt that they were rewarded appropriately, but the wording here seems to almost deliberately stoke the developers vs. management flames, especially coming from a VC...
I'm glad that ycombinator isn't listed as the source, at least. I think it would be fairly interesting to see what kind of data they have about various social medias and link propagation.
And also, a slight sickening sensation that marketers are doing this. Not that I'm surprised, it's just that I'm sure they have papers written saying that "Twitter has a 64% link click rate if you write your post like X, whereas Facebook can achieve as high as 70% if you do Y."
I know it has been going on for years on the web, and decades for other marketing, but here it's plain to see. It fills me with unease, like I'm no longer in control and what I like in life is already planned out.
This is a pretty standard Google Analytics Tracking URL [1] - setting aside the Googleness of it (which raises a different set of issues) I look at it like the modern equivalent of an Apache Access Log.
Also, not sure if this will make you feel better, but most places that are doing this type of in-depth analytics, etc. are much more often trying to bend themselves to your will ("what feature, benefit or product that people will get value out of") than to trick you into liking something you otherwise wouldn't.
One way of interpreting that query string is as the answer to an implicit survey question of "How would you like to get the stories we write?" the answer in this case being: "Twitter" [2].
Incidentally, this is why I recommend rewriting URLs to hide UTM strings on load: http://esd.io/blog/stripping-utm-strings.html This way they 1) look cleaner to the user (of a modern browser) and 2) if they are reshared it doens't come with a inaccurate tracking code.
those are standard google analytics campaign tags, they probably (automatically?) built the link that way before they originally shared it on twitter, and then someone cut & pasted that link to share on HN.
There's a fix for this that the HTTPS site can add to <meta>, which works with newish browsers: the "meta referer" header: http://wiki.whatwg.org/wiki/Meta_referrer
This is great advice even for students because sometimes the things that younger people (anecdotal: me) give a lot of importance to and feel terrible for screwing up end up not mattering at all.
1. Things like validation from peers / popular kids
2. Screwing up on that major test that sometimes "defines" where you end up. A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly).
3. The academic thing is worse if you are artistically inclined and have science and math thrown down your throat telling you that humanities are for those that can't cut it in science (I know friends that have had a variation of this conversation with their kin/parents, etc) quite true in countries like India / Pakistan, Korea and perhaps even China.
4. Humiliation/shame from something that seems drastic in the moment (Getting rejected by the cute girl in front of all your friends and classmates. Yea that sucks). Getting labeled sucks too, a lot of girls can be mean to their own gender at that age which is absolutely disgusting.
Unfortunately for most people it is something that they only figure out in hindsight.
I had an Economics professor who gave a small lecture to our class after we got our midterm grades back.
"This midterm is one third of your grade. This class is one of four you are taking this semester. This semester is one of eight you will take at $University. And while people will tell you otherwise, your time in college is a small part of your life. It will be okay."
$ A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly).
Heavy debts are a significant contributor to suicide rates, fear and shame play a large part. The potential collapse of companies would heighten this risk, so here's some statistics on the Great Depression (bottom of the page):
This is not true. It's almost always true in most contexts, but to say that it's a fact is to encourage a false sense of security... sometimes things are that bad, and it's important to be able to realize and accept that, otherwise you never even give yourself the option to try and improve them.
Just the point I came to comment on. There have been many times in my life where things seemed to be completely falling apart. But I have learned that I almost always paint a much bleaker picture than what actually ends up happening. Life goes on.
The most interesting take away from that was how resilient companies appear to be despite their leaders best efforts to sabotage everything. I think the real lesson is (and I don't like it): if you want your company to survive you need to be good at spin.
In the case of hotmail, how did he ever let a situation arise where one bug in a nightly cron script could obliterate his users emails?! No backups because they cost too much?? "the sun, the moon and the stars lined up against us for it to happen" No, you were just reckless and too busy chasing growth.
With Ironport, he says his whole business was built on the back of anti-spam yet they were totally unprepared when their partner who provided the anti-spam tech they used pulled the rug from under them (despite knowing full well this day would come). The real lesson here should be: try not to outsource the key component upon which your entire business depends.
I had to laugh at the end when he says the only time he's had a failure was when he was using his own limited resources to fund a business and not investor money. Well yes... mistakes are easier to hide/absorb when you're swimming in cash. Not so much when things are tight and you can't afford to learn after the fuck up.
Both Hotmail & Ironport reacted with a customer-centric focus (e.g. Hotmail: "How do we recover their emails? What else can we provide customers in the future? What steps will we take so this never happens again?")
I would guess a lot of companies in a WFIO situation that take a CYA approach rather than customer-first face a worse fate.
Question - why did IronPort cancel the contract with their customers if they just got the feature implemented that they seemed to want? Was it a done deal - i.e. they were losing those customers anyways?
Also, survivorship bias much? Two examples --> conclusion? Was nice hearing about those company's histories though, didn't know about the Hotmail thing.
"Instead of Symantec canceling the contract, we went on the offensive and faxed a letter to all of our customers cancelling the contract with THEM [Ed.:SYMANTEC]– a position of strength."
I initially misread that too. He's actually saying they faxed a notice to their customers that they were canceling the contract with Symantec. It's worded poorly.
The same principles were true and applicable in a much larger near-implosion that I experienced from 2008-2012 working for a "too big to fail" financial company. Strong, out-front leadership, a strong team led the way. It took time for us to realize that it wasn't as bad as it seemed. We had to learn that the external pressure from the media, public and even friends and family was only a distraction.
In this situation it was quickly apparent that my role was to focus on a solution, not dwell on the problem or the cause. This should be true for everyone on the team, unless you are the specific individual who caused a global financial meltdown, deleted all the email or caused a critical failure.
Actually that was a problem due to the culture of stupidity and willful ignorance of risk, not some keyboard mash that caused the markets to crash.
It might be one of the rare situations where it was so bad, that after stabilizing the situation, many of the responsible parties needed to be put down.
I know from my experience working with Wachovia, the gross incompetence, system wide narrow minded greed and the fuck-everyone-but-me attitude was so bad it was mind boggling.
It was not a mistake, this was a concequence of the culture and goals of those companies.
It took time for us to realize that it wasn't as bad as it seemed.
I'm not sure this example is relevant, since this realization would not have been possible without a massive leg-up and pat on the head from your national government.
That is: "too big to fail" negates "near-implosion."
I really admire Scott from what he's written. He can be an inch close and still get out of WFIO situation. And he seems to know the difference when it's doomed like with e-commerce business. So keep rowing, Scott!
I am disappointed to see the top comment here criticizes this as "survivorship bias." Maybe the survivors have something useful to communicate that can help others survive what looks "impossible" when you first run into it? Maybe that's the point?
Though this probably shouldn't be exactly a surprise to me given how much shit people give me any time I try to talk about getting well after doctor's basically wrote me off for dead in some sense. No one wants to learn from that either and I honest to god don't get it.
Great points, especially leading from the front when times are tough. I didn't think Hotmail would have survived the outages, but they came out just fine.
I particularly agree with the point on leadership. If you can motivate those around you, people perform so much better and can sometimes come up with better solutions. Doesn't always guarantee you coming back from the brink, but stands you in good stead.
"We're Fucked, It's Over" strikes me as a pretty dramatic response to a technology vendor failing to renew a contract. Unless they have extra-special access to customers or customer data, whatever they do can be replicated in-house.
Fascinating to read a 1st person account of a near-death moment. "Companies are damn resilient" - I think entrepreneurs are damn resilient, too, for making it through crises like this over & over...
My palms got sweaty just reading the hotmail portion. I think someone could make a really great horror book that is just a collection of stories like this. I know it would give me nightmares.
if it's never as bad as it seems: does this make a case for optimism or realism on a team?
regardless, that's precisely why you need a cofounder who balances you. whether it's a different perspective, questioning some of your choices, or holding you accountable (and holding you up): you need it. you'll make an order of magnitude more poor choices on your own.
The leader needs to be the first one there, the last one to leave, and be willing to do anything it takes – like answer customer care calls or personally drive a replacement part to an irate customer. Nothing is beneath a leader in times of crisis.
I like this, because it says two things at once, both true.
First, if you're in a position of ownership/authority but you're not willing or able to do the grunt work, you won't have the credibility for long. People will follow orders out of fear of getting fired, but you'll never get more than the bare minimum. There's a point where leaders are replaced or outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites) and after that point, the organization can't even motivate shit to stink.
Second, it might be that it's just not worth it to lead from the front. That means that you're not really in a position of leadership. You might be a middle manager who realizes that the people above you will never buy in to what you're doing. Then you'll probably lose that desire to make those sacrifices. That's fine. You shouldn't tie yourself to the mast, at that point, because you're not really in a leadership role anyway.
http://www.fastcodesign.com/1671172/how-a-story-from-world-w...
"In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."...
"[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes."
Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional.
For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it.
http://www.aaronkharris.com/when-seo-fails-single-channel-de...