
Ask HN: What is your failure story? - codesternews
We hear success story all the time. That is good but I think failure stories can give more learning.<p>What is your business or startup or carrer failure story?
======
ideonexus
I was responsible for a software bug that forced the entire United States
Coast Guard to revert to using paper records to track all of their Aviation
Logistics for an entire month. The bug was in an aircraft maintenance
dashboard that had over a dozen filters on it. It passed a thorough round of
QAing, but once in production it was found that a specific selection of
filters would create an infinite loop in the report that would crash the
system. With hundreds of mechanics using the dashboard, someone was triggering
that selection of filters every two to three days.

It was a very rough month for me. When an emergency patch failed to solve the
issue, management decided to take the whole system down (it was too big a
release and had been in production too long to revert). User satisfaction
surveys bottomed-out. There was discussion in upper-management about firing
me, but my immediate supervisor defended me and pointed out that this required
failure at many levels. When we got the system back online, they had to hire
dozens of contractors to manually enter all the paper records into the system.

I continued to code for the Coast Guard for five more years, got a big
promotion to Senior Developer at one point, and left the organization with
much respect. Every once in a awhile someone would present a graph of user
satisfaction surveys over the years and there would be that awful month where
the graph bottomed out, but I could eventually smile about what that junior
developer did.

~~~
kadendogthing
Huge kudos to your manager for standing up like that. Companies often look for
a single point of failure (it's only human), instead of admitting that the
organization as a whole systematically failed to prevent the error, let alone
discover it "in time."

I hate to use platitudes but companies are only really as strong as their
weakest link (not to impugn your skills or anything like that). Letting a
single failure sink the whole ship means there are larger issues at play.

~~~
crimsonalucard
You also need to be a person worth fighting for. It goes both ways.

~~~
kadendogthing
Presumably that's what the hiring process is for.

~~~
crimsonalucard
Right. Everyone knows that the hiring process is perfect and accurate and that
if you're hired there's no need to prove yourself because you already did that
in the interview are you kidding me?

~~~
kadendogthing
I think you have a very maligned view of what managers should be if you agree
with the thought that people need to prove themselves to managers to be
"fought for." Especially over a mistake that multiple layers of the
organization didn't catch and weren't equipped to handle.

No I'm not kidding you. That view point seems either very naive or
nonconstructive. And considering you had to resort to a disingenuous straw man
in order to respond to something I didn't say or imply, I'd say the latter is
more likely.

~~~
crimsonalucard
You need to constantly prove yourself everyday. Lets put it this way. If I
don't show up for work and I did well on the interview. I'm not proving myself
aren't I? If I do horrible work, if my code is buggy and full of errors then
what am I?

No manager will fight for that despite a strong interview. This is simple
logic and common sense.

Obviously the person in the original posting proved himself.

------
ekns
Failure 1: I'd planned to do a PhD eventually, but I ended up more or less
self-sabotaging my degree grade by being depressed and not doing anything
about it (2:2 in the UK system vs. the nominal 2:1 ostensibly required for
MSc/PhD applications to most places)

Failure 2: For the longest time, I never managed to make a good routine for
myself. To learn, build things, exercise, meditate, etc. Good habits seem to
be a pre-requisite for success in virtually any endeavor.

Failure 3: I had an interview scheduled with Google once, right after I had
finished my military service in Finland (it's more or less mandatory for men).

I had scheduled it a tiny bit too close for comfort, just after I was supposed
to have been out of the garrison. Turns out they called an hour too early (I
specified UTC timezone but they marked it down as BST..) so I could only
awkwardly tell them I could not talk at the time, and ask if they could call
again an hour or a day later.

I never could get another call and settled for a crap job (by some standards).
Maybe I could have earned a lot more and learned a lot more (and had a better
network of contacts) if things had turned out differently :p

On a more positive note, I started doing freelancing/consulting last year
after "no one" seemed to want to hire me at a decent salary and proceed to
make OK amounts of money through random gigs.

Still figuring out how to network effectively. I figure the best bet is
through gaining visibility online by publishing something useful.

~~~
Aleroniponi
I am in a similar predicament currently, and just want to let you know that
there are others whom share your pain(s). I am 23 in 10 days, and haven't been
in education since my community college days about 4 years ago. I have felt
like a worthless pile of rubbish since then, however it is all related to
depression, anxiety, ADHD, and an unsupportive family.

Personally, I find that small, but actionable changes to your routines and
habits make the biggest difference. Please hang in there man!

------
itamarst
* First month on the job I wrote some software that took our biggest customer offline every night (would've been during the day too if I'd picked a smaller constant in my code).

* Accepted a job for vastly less than market rates, luckily realized before too late.

And honestly I make more mistakes every week... Full versions of above stories
at
[https://codewithoutrules.com/softwareclown/](https://codewithoutrules.com/softwareclown/)

~~~
m0ck
Can confirm, I'm subscribed to Software Clown for almost a year now and still
have fun reading it every week (and I like to think I've learnt something from
it as well). I love how you keep the articles short and dense so is no problem
to read them even standing in crowded bus :)

------
tracer4201
Not a complete failure story, but I joined a FANG company as an entry level
engineer. In about three and a half years, I’ve gone through two promotions.
My most recent team where I got promoted to senior engineer is demoralizing to
me and gives me anxiety even thinking about work.

I’m feeling a mixture of burnout and just a general disinterest in my teams
domain area. My current team has also grown in ways I didn’t expect when I
joined.

My coworkers don’t seem the most motivated. I spend less time coding or
building anything and more time sending status reports and pointless meetings.
I worked on very impactful things early in my roles at this company. I
question the value of things my team is doing now.

My direct manager is very passive. There’s literally no feedback or much
engagement from him. His job is to make sure sally doesn’t stab sue kind of
thing.

Ironically, the culture is very cut throat. Longer term, I’m realizing I will
have to quit and find a job elsewhere. It’s a bit demoralizing after earning
two promotions and putting in as much as I have. My husband and I were
planning on getting pregnant by next year and purchasing our first home. If I
leave my job, we have a sizeable nest egg but both plans will be up in the
air. I haven’t said anything to my husband yet.

I still have a job, but I’m questioning what was all this for.

~~~
deanmoriarty
Thanks for your post.

I am currently at a startup and have been here a few years. I came to really
hate the environment, mostly because the people this company can hire are not
what I would personally consider talented (due to a mix of non-exponential
company growth and lower compensations than FAANG companies). The result of
hiring low quality professionals is that the engineering standards are so low
that it really depresses me.

For this reason, I've interviewed and got offers with 3 FAANGs, and I am
contemplating about moving to one of them. After reading your post, I
recognize I am perhaps romanticizing FAANGs a little bit more than what's
fair, but the thing that intrigues me the most is the ability to work with and
learn from other very talented engineers (so I can grow), that I have a hard
time finding in startups that don't have big pockets as them.

And of course, also the financial aspects, since financial freedom is
relatively high in my priority list.

~~~
souprock
FAANG companies are not where you go to find high engineering standards. You
need to look for a job featuring one of these:

DO-178C DAL A (if you mess up, aircraft crash)

EAL5, EAL6, EAL7 (if you mess up, national secrets leak)

MISRA C (if you mess up, cars crash and a giant recall is required)

IEC 62304 Class C (if you mess up, medical devices kill people)

NUREG/CR-6463 (if you mess up, you get a nuclear disaster)

ISO 9000 and ISO 9001 (if you mess up, it will at least be documented)

If you didn't really mean high engineering standards, and you just want to
work with capable people, you can find that all over the place. Perhaps being
within commute distance of a FAANG has an impact, and you are seeing that due
to your location. There are many people who refuse to live anywhere near a
FAANG and/or wouldn't tolerate the culture.

~~~
deanmoriarty
Yeah, I definitely didn't mean high engineering standards then. The places you
are listing wouldn't work for me because of the little financial incentives
they provide. As far as I know FAANG are the only places where a reasonably
decent software engineer can make ~$400-500k/y (at least those were my
offers), and by living frugally the way I do that means massive savings even
in a high cost of living location, which puts me closer and closer to my
financial freedom goals they day I decide to be done and move to a low cost
area.

So, I guess working with "very capable people" and a very good paycheck is
enough for me :-)

~~~
souprock
Do you have a well-defined goal in terms of dollars and years? If not, you
might always want a bit more money, and thus never quit. You'd miss out on the
chance to raise a family. It's safest to get that done before age 30. No
matter how much money you get, you can't buy back your younger years after
they are gone.

~~~
deanmoriarty
Yes, if I can continue with my current savings rate, I should be done by my
mid-late 30s (very early 30s now), with my passive investments fully covering
my cost of living at least in Mediterranean Europe (where I'm originally from)
or South East Asia (where my partner is from), both amazing places to live :-)

Both my partner and I are not looking to have kids (we've thought about it for
many years before reaching the conclusion) and we already have extended
families with tons of kids, so from that point of view we feel there's nothing
we're missing, it's a non-priority.

For everything else, I'm not exactly wasting my life right now, I'm enjoying
it even in a very high cost of living area: if you can deal with housing (and
not having kids admittedly helps a lot since you can live in way smaller and
cheaper places close to work), things are not that much more expensive than
anywhere else, so I get a lot of experiences with good arbitrage :-)

------
MivLives
Burned out six months into my first post college job. Not as dramatic as some
of the other failures here. Just stopped showing up, and refused to leave my
room for a week. Eventually got up the nerve to call them and explain what
happened. They were willing to take me back but I realized it'd likely happen
again if I continued.

So I changed fields inside tech, from IT support/sysadmin to web development.
On the job hunt right now so still not sure if it was the right choice. I have
been enjoying learning it significantly more than the sysadmin stuff.

------
lelima
Once I scale up Azure analysis services(main data source for company Marketing
and decision making) cause we were running low memory in peak hours, only to
find out that I couldn't scale down, increasing cost by 300%. I had to
recreate all the models again in the old tier increasing even more the cost
cause we have to maintain both data sources till I finish with the creation of
the models. My boss was mad.~azure lvl 100 destruction~

~~~
tabtab
Here's my most memorable Microsoft snafu: My desktop was just upgraded to
Windows 10. The "Favorites" links in File Explorer were converted into "Quick
Access" links by the Microsoft migration process, but the aliases (titles) of
the links had changed, or so I thought. I renamed the aliases back to what
they were under Windows 8.

The phone started ringing with complaints, and to my horror I learned that I
inadvertently _renamed_ widely-used WAN and intranet folders. Quick-Access had
no aliases: I had inadvertently changed the _actual_ folder names. A colleague
walked by and asked, "Why are you sweating and white like a ghost?"
Fortunately I renamed them back before too much damage was done.

------
tabtab
In my first job out of college, I had no training or experience in customer
service. I was horrible at it without knowing what I was doing wrong.

For example, our system allowed for different ship-to's per order, but one
vendor's receiving system wasn't able to handle multiple ship-to's. I should
have referred them to my supervisor to negotiate something, but instead said,
"you'll just have to change your system." They got irate. That and a few other
"people skills" issues that needed tuning got me FIRED. I was devastated.

I had taken a "communications" course in college, but it was horrible in
hindsight. The author and instructor should have been fired...uh, mentored, I
mean. They should have made us read, "How to Win Friends and Influence
People". Colleges have something against that book because it's not based on
empirical data and doesn't try to sell the idea of using good logic. Instead
it deals more with the "squishy" side of human nature. Humans are social
creatures above all else.

------
tracker1
Once worked for a company that had a subscription based service. They wanted a
monthly report of expired accounts so that they could warm-call for renewals.
The bug was that some accounts had paid for 2 years. After a couple years,
someone noticed that the reports hadn't been including the 2-years
subscriptions and it was effectively a loss of a couple million in revenue.

While I did code it to the specifications I was given, and was really clear on
understanding it was still a pretty big failure.

\-----

Second biggest failure was checking in the db connection against a production
database into source control. It was internal source control, and the
transforms (ASP.Net Web.config transforms) had it anyway, so wasn't an
increased risk in that regard. I was troubleshooting a production issue.

What happened is another developer had used that connection configuration to
drop/recreate a table. It took a few hours to recover from (DB restore, then
replay against API activity logs). And a day of lost time for a few hundred
workers.

------
adamredwoods
In the pre-HTML5 days of the Internet, I wanted to be a Flash Designer.
Forever. I loved the platform and process so much, that I only had stars in my
eyes for Flash! The idea of being able to blur the lines of animation,
programming and design was wonderful. Then it died (April, 2010
[https://www.apple.com/hotnews/thoughts-on-
flash/](https://www.apple.com/hotnews/thoughts-on-flash/)) and so did my
career.

Fortunately, in most failure stories we can find some ways to recover. Knowing
Actionscript developed some programming skills, which were applicable to
HTML5.

My key lesson is learning how to pivot from one technology platform to
another. It's not always easy, requires some humility, but it's possible.

------
thewizardofaus
I developed an app many years ago , helping students find cheap eats around
campus. The traction for people to use it was their but I didn't plan for the
food stall owners to be extremely stubborn with prices and deals.

~~~
techsin101
What happened?

------
muzani
0\. Graduated from a top CS uni. Starting salaries were lower than KFC
assistant manager. Started looking outside tech to for enough income to
sustain being a programmer.

1\. Put down $6000 on a coffee kiosk. Made real good money every day. But
partners were all dishonest and manipulative.

2\. Built an app for a guy. He paid well. Built another app for him. Contract
sucked, one clause meant we couldn't get paid even though the project was 75%
done.

3\. Wedding planner business. Too much work, too little profit.

4\. Brownies dropship, successfully distributed in petrol stations. Slow cash
flow, dishonest suppliers.

5\. Sold snacks and tarts. Did very well in festive season, but demand
flatlined after.

6\. Manufactured crab crackers. Unit economics didn't work out. Dishonest
suppliers. Too capital intensive.

7\. Did prototype for a big company. Contact person later paid me half of what
he verbally proposed. Project had potential, but felt cheated.

8\. Large tech project. Contact people were aggressive negotiators, trying to
get insultingly unfair deals. One partner ragequit over it so we didn't pursue
it.

9\. Tried to do price comparison startup for groceries. Grocery suppliers
didn't like it because it puts pressure on their low margins. Gov warned us
that crowdsourcing is illegal if we didn't get grocers' consent.

10\. Pivoted to recipes + e-commerce. Went remarkably well, but had to do some
unethical things.

11\. Tried to pivot to Blue Apron style service. Too capital intensive, not
enough VCs here. Logistics and market were underdeveloped. (A YC startup ended
up doing this reasonably well, Dahmakan, but with substantial changes)

* Success point! Gave up on the startup. Sold whatever I had of my startup at this point, at enough money to cover up for all the above losses. Bought an Alienware and 4K monitor, wasted 1-2 years of my life playing games.

12\. Built app for a guy. Was paranoid of payment so there was a standoff
where I refused to add features and fix bugs until client paid. Project was
90% done but client didn't close deal because of bugs.

13\. Took on a hell project just to try to be a hero. Unreasonable budget,
unreasonable specs, nobody wanted the job. Couldn't do it.

14\. Worked on a startup for a guy. Too ambitious, architecture was too
complex for team. Employees were overworked and unfocused, often "forgetting"
to do things they were supposed to. They brought in consultants, managers,
interns. We had 2 day sprints where half the time was spent in meetings
discussing timeline.

15\. Inherited company from late father as a non-executive director, hoping
for passive income. A Fortune 50 company didn't pay, putting said company
about $500k in debt/bankruptcy. Saw the red flags early, but hard to put foot
down on family business.

16\. Major project for multinational companies (plural). Top notch team. But
VP insisted on unreasonable deadlines. 6 month project shortened to 6 weeks,
at half the quoted price. Hacks were made, failed quality control. Teams
blamed each other, started wasting each other's time to buy more time for
their own work.

It seems like a lot but there's been some moments of success as well :)

~~~
allwynpfr
Duuudeeeeee this was like watching a movie! You should totally turn this into
a blog, perhaps even a book I bet you learnt lots along the way, you're
literally the poster boy/girl for try, fall down, stand up, try Thanks for
this :)

~~~
muzani
Yeah I can see how some people would get discouraged. But I grew up playing
roguelikes. It was normal to try something, fail, then try again,
understanding why one approach didnt work. It's actually fun to start
something after a previous failure, because you'll be excited to try the path
not taken.

If anything the mistakes I do regret is trying to settle for a comfy corporate
job after selling my company. The failures were out of my control, and there
was little learned from it.

------
pipelineist
Failure 1: Worked for a friend's company. As employee #1. I was to have all
kinds of freedoms and responsibilities. Sounded really awesome.

Failure 2: Was too conflict-shy to draw firm boundaries, and likewise was he.

This friendship of 20 years is no more.

(While working for said friend) Failure 3: Offered a hair-raisingly badly
thought-out fixed-price bid. Both my friends's mistake and mine. We probably
sunk about 5x what we stood to make from it, and then abandoned the project,
embarrassing a relative in the process.

There's plenty more of course, but these bother me.

------
otohp
Me - a graduate student in a university. Looking around on all servers on the
network and found a program called "crack". it was designed to reverse-
engineer passwords given a unix passwd file. I thought, what the heck, let me
see if I can get some gems. SO I used yppasswd to download all password
hashes, and ran crack on it. Needless to say, sysadmin found out in a couple
of days, and suspended my account for 2 weeks :(

------
nbardy
Working remotely and contracting so I could travel the world. Went well until
I burned out and ended up a broke alcoholic.

------
harinik
Me and my friend developed the a call center on the cloud software
(contactcentral.io) .We feel product is on par with other competitors in the
same space but we failed in marketing. We both came from engineering
background bootstrapped ourselves, couldn't find better marketing strategy to
position it properly by generating actual leads.Either we should have cash to
spend on marketing or good minds working on marketing.

------
cattlefarmer
Back in uni, I wanted to be a motorcycle designer. I managed to get into Honda
Japan's internship program, which was essentially a 3 day long interview
process. They give you some design specs on the first day and then lock you in
a room with 10 other kids to draw motorcycles non stop for three days. 4 years
of effort all came down to these three days, and I couldn't draw anything on
the first day. My mind went blank. I stayed up all night to catch up on the
drawing. That was my first failure.

The lack of sleep meant I couldn't focus much during the second day. I pushed
through and kept drawing. My next failure that night was to again forgo any
sleep just to keep pushing out designs. On the third day, I was so exhausted
by the lack of sleep, all the stress I piled on myself after all those years
of expectations, My mind went numb during the morning's final prep. I was so
tired, I didn't even know what the hell I was talking about during the final
presentation, I interrupted the judges, I talked back, I argued. Naturally, I
failed to progress to the next stage.

That failure was the beginning of a series of failures over the next several
years.

I broke up with my fiancee to 'focus on my studies/career'.

I couldn't get a product/industrial design job because I was so focused on
motorcycles during my studies that I was fairly terrible at most other
products. Desperate for any job, I become a web designer for a company on the
other side of the country. The job lasted six months.

I wasted money moving again to another city to search for better opportunities
but found little. After another six months, I was out of savings and my family
wouldn't lend me anymore money. I gave up, I contemplated suicide for a while,
but finally moved back home to work for a relative.

About a year in, I got together with a couple of friends to set up our own
design agency. One of them, talked the big talk but spent all his time playing
games. Tried to rush work out the day before the deadline. The other worked
really hard but wasn't particularly talented and decided to go back to his old
job. I went back to work for my relative again.

After several years, found a small dev shop willing to take me in for not very
much money. Spent two years there.

I'm in a better place now but those six years were terrible and whenever I
look back at it, I've always felt that first failure at Honda just set a lot
of things in motion. I was burnt out and depressed but I wouldn't admit it, so
it just kept getting worst.

I have much gratitude for that relative who put up with me the whole time and
gave me the opportunity, namely time to heal and a place to rest, to turn
things around.

------
cableshaft
Failure #1: Joined a mobile app startup as employee #3 for a lot less money
than I should have, run by a guy who really didn't know how to run a startup,
he had just convinced the parent company to give him some money to make mobile
apps. He couldn't raise money, only get handouts from the board. I falsely
assumed because those people were rich as hell and the other companies they
owned were successful that we'd be okay also. But they didn't want to spend
hardly any money, especially on marketing, and when our apps failed to make
money, it just convinced the board to pull the plug on us.

Which lead to Failure #2: The startup guy decided to make a couple apps as
'warmups' before his main app idea. When those apps were released and didn't
make money, the board became more wary about letting us do the 'main' app. If
we had just started with the main app to begin with, they wouldn't have seen
those app failures and we could have probably convinced them to keep funding
it until release and that app might have actually seen success. Having low-
investment but visible failures made the company look like a failure, and
insured its death.

But not a failure: I had a lot of autonomy, I had my first experience managing
other employees, and the apps we worked on helped me learn a ton in a short
amount of time, and I'm still proud of them years later, even if they didn't
reach much of an audience. That experience also helped me land my next job.

I could probably do that about every job in my career, honestly. Pretty much
every job I've had was a mistake and a valuable learning experience at the
same time, at least up to a certain point. Before this job I had joined
another startup as employee #3 that failed for different reasons, and after
this job I joined a small company that ended up having its worse year ever and
lost all of their clients, and then I took a contract job I despised just to
keep afloat for awhile where the entire department fell apart when the lead
dev left and they terminated my contract, but that gave me the skills that
helped me get my current job I've been at for 3 and a half years which started
so successful it was bought by a larger company, and has been very slowly
eviscerated by that larger company as it has been forcibly integrated into the
rest of its corporation (our company, now division, has about 15% of the
workforce it had when it was purchased, and it had over 500 employees a the
time), and yet I've learned a lot while I've been here and worked on projects
for the most highest profile clients in my life (you can't get higher profile
than them, really) and for the first time I've actually had to really, really
care about optimization since our traffic is massive and constant, although
the stress has gotten beyond ridiculous now and I need to leave soon.

------
notafrog
Published an app on the App Store that tried to connect to localhost instead
of the production API server...

