Hacker News new | comments | ask | show | jobs | submit login
Ask HN: What is your failure story?
61 points by codesternews 3 days ago | hide | past | web | favorite | 45 comments
We hear success story all the time. That is good but I think failure stories can give more learning.

What is your business or startup or carrer failure story?

I was responsible for a software bug that forced the entire United States Coast Guard to revert to using paper records to track all of their Aviation Logistics for an entire month. The bug was in an aircraft maintenance dashboard that had over a dozen filters on it. It passed a thorough round of QAing, but once in production it was found that a specific selection of filters would create an infinite loop in the report that would crash the system. With hundreds of mechanics using the dashboard, someone was triggering that selection of filters every two to three days.

It was a very rough month for me. When an emergency patch failed to solve the issue, management decided to take the whole system down (it was too big a release and had been in production too long to revert). User satisfaction surveys bottomed-out. There was discussion in upper-management about firing me, but my immediate supervisor defended me and pointed out that this required failure at many levels. When we got the system back online, they had to hire dozens of contractors to manually enter all the paper records into the system.

I continued to code for the Coast Guard for five more years, got a big promotion to Senior Developer at one point, and left the organization with much respect. Every once in a awhile someone would present a graph of user satisfaction surveys over the years and there would be that awful month where the graph bottomed out, but I could eventually smile about what that junior developer did.

Huge kudos to your manager for standing up like that. Companies often look for a single point of failure (it's only human), instead of admitting that the organization as a whole systematically failed to prevent the error, let alone discover it "in time."

I hate to use platitudes but companies are only really as strong as their weakest link (not to impugn your skills or anything like that). Letting a single failure sink the whole ship means there are larger issues at play.

Re: Huge kudos to your manager for standing up like that. Companies often look for a single point of failure

Indeed. Orgs like simplistic stories and simplistic blame (scapegoats). They'd like to fire somebody(s) and say, "we solved it all." Anything important/critical should have multiple eyeballs vetting it, and mistakes do happen. If they don't have multiple eyeballs vetting it, then the people who failed to hire enough should be given a stern warning.

I don't believe in firing for one mistake, unless it's egregious. People get better as they learn from mistakes.

`Work for people who can put trust in you, and can fight for you. ` a lesson I learn the hard way as well

You also need to be a person worth fighting for. It goes both ways.

Presumably that's what the hiring process is for.

Right. Everyone knows that the hiring process is perfect and accurate and that if you're hired there's no need to prove yourself because you already did that in the interview are you kidding me?

I think you have a very maligned view of what managers should be if you agree with the thought that people need to prove themselves to managers to be "fought for." Especially over a mistake that multiple layers of the organization didn't catch and weren't equipped to handle.

No I'm not kidding you. That view point seems either very naive or nonconstructive. And considering you had to resort to a disingenuous straw man in order to respond to something I didn't say or imply, I'd say the latter is more likely.

You need to constantly prove yourself everyday. Lets put it this way. If I don't show up for work and I did well on the interview. I'm not proving myself aren't I? If I do horrible work, if my code is buggy and full of errors then what am I?

No manager will fight for that despite a strong interview. This is simple logic and common sense.

Obviously the person in the original posting proved himself.

Wow...I can't even begin to imagine how awful you must have felt. But, in hindsight, it's really cool too, in a weird way.

Failure 1: I'd planned to do a PhD eventually, but I ended up more or less self-sabotaging my degree grade by being depressed and not doing anything about it (2:2 in the UK system vs. the nominal 2:1 ostensibly required for MSc/PhD applications to most places)

Failure 2: For the longest time, I never managed to make a good routine for myself. To learn, build things, exercise, meditate, etc. Good habits seem to be a pre-requisite for success in virtually any endeavor.

Failure 3: I had an interview scheduled with Google once, right after I had finished my military service in Finland (it's more or less mandatory for men).

I had scheduled it a tiny bit too close for comfort, just after I was supposed to have been out of the garrison. Turns out they called an hour too early (I specified UTC timezone but they marked it down as BST..) so I could only awkwardly tell them I could not talk at the time, and ask if they could call again an hour or a day later.

I never could get another call and settled for a crap job (by some standards). Maybe I could have earned a lot more and learned a lot more (and had a better network of contacts) if things had turned out differently :p

On a more positive note, I started doing freelancing/consulting last year after "no one" seemed to want to hire me at a decent salary and proceed to make OK amounts of money through random gigs.

Still figuring out how to network effectively. I figure the best bet is through gaining visibility online by publishing something useful.

I am in a similar predicament currently, and just want to let you know that there are others whom share your pain(s). I am 23 in 10 days, and haven't been in education since my community college days about 4 years ago. I have felt like a worthless pile of rubbish since then, however it is all related to depression, anxiety, ADHD, and an unsupportive family.

Personally, I find that small, but actionable changes to your routines and habits make the biggest difference. Please hang in there man!

* First month on the job I wrote some software that took our biggest customer offline every night (would've been during the day too if I'd picked a smaller constant in my code).

* Accepted a job for vastly less than market rates, luckily realized before too late.

And honestly I make more mistakes every week... Full versions of above stories at https://codewithoutrules.com/softwareclown/

Can confirm, I'm subscribed to Software Clown for almost a year now and still have fun reading it every week (and I like to think I've learnt something from it as well). I love how you keep the articles short and dense so is no problem to read them even standing in crowded bus :)

Not a complete failure story, but I joined a FANG company as an entry level engineer. In about three and a half years, I’ve gone through two promotions. My most recent team where I got promoted to senior engineer is demoralizing to me and gives me anxiety even thinking about work.

I’m feeling a mixture of burnout and just a general disinterest in my teams domain area. My current team has also grown in ways I didn’t expect when I joined.

My coworkers don’t seem the most motivated. I spend less time coding or building anything and more time sending status reports and pointless meetings. I worked on very impactful things early in my roles at this company. I question the value of things my team is doing now.

My direct manager is very passive. There’s literally no feedback or much engagement from him. His job is to make sure sally doesn’t stab sue kind of thing.

Ironically, the culture is very cut throat. Longer term, I’m realizing I will have to quit and find a job elsewhere. It’s a bit demoralizing after earning two promotions and putting in as much as I have. My husband and I were planning on getting pregnant by next year and purchasing our first home. If I leave my job, we have a sizeable nest egg but both plans will be up in the air. I haven’t said anything to my husband yet.

I still have a job, but I’m questioning what was all this for.

I am in a very similar situation, 4.5 years in at a large / prestigious organization where I was thinking there was a lot of scope for progression and I could stay long term.

First 3 years were great, good work and appraisals, I learnt a lot. After that, things deteriorated quite suddenly, a combination of some urgent but poorly conceived projects, passive management, too much time in meetings, a few layers of extra paperwork / approvals introduced.

Basically, the work environment really did go downhill, but in hindsight I didn't deal with that very well or put together a plan to move.

I would suggest you come up with a plan to bring this to a head sooner rather than later. Try to do an excellent job with your status reports and pointless meetings, so that you can transfer or get a good reference from your manager, and leave on a high note. At the same, research the market for your next job, pick up any missing skills, and prepare for a move on your own terms and schedule. Don't let the situation fester without doing anything about it.

Thanks for your post.

I am currently at a startup and have been here a few years. I came to really hate the environment, mostly because the people this company can hire are not what I would personally consider talented (due to a mix of non-exponential company growth and lower compensations than FAANG companies). The result of hiring low quality professionals is that the engineering standards are so low that it really depresses me.

For this reason, I've interviewed and got offers with 3 FAANGs, and I am contemplating about moving to one of them. After reading your post, I recognize I am perhaps romanticizing FAANGs a little bit more than what's fair, but the thing that intrigues me the most is the ability to work with and learn from other very talented engineers (so I can grow), that I have a hard time finding in startups that don't have big pockets as them.

And of course, also the financial aspects, since financial freedom is relatively high in my priority list.

FAANG companies are not where you go to find high engineering standards. You need to look for a job featuring one of these:

DO-178C DAL A (if you mess up, aircraft crash)

EAL5, EAL6, EAL7 (if you mess up, national secrets leak)

MISRA C (if you mess up, cars crash and a giant recall is required)

IEC 62304 Class C (if you mess up, medical devices kill people)

NUREG/CR-6463 (if you mess up, you get a nuclear disaster)

ISO 9000 and ISO 9001 (if you mess up, it will at least be documented)

If you didn't really mean high engineering standards, and you just want to work with capable people, you can find that all over the place. Perhaps being within commute distance of a FAANG has an impact, and you are seeing that due to your location. There are many people who refuse to live anywhere near a FAANG and/or wouldn't tolerate the culture.

Yeah, I definitely didn't mean high engineering standards then. The places you are listing wouldn't work for me because of the little financial incentives they provide. As far as I know FAANG are the only places where a reasonably decent software engineer can make ~$400-500k/y (at least those were my offers), and by living frugally the way I do that means massive savings even in a high cost of living location, which puts me closer and closer to my financial freedom goals they day I decide to be done and move to a low cost area.

So, I guess working with "very capable people" and a very good paycheck is enough for me :-)

Do you have a well-defined goal in terms of dollars and years? If not, you might always want a bit more money, and thus never quit. You'd miss out on the chance to raise a family. It's safest to get that done before age 30. No matter how much money you get, you can't buy back your younger years after they are gone.

Yes, if I can continue with my current savings rate, I should be done by my mid-late 30s (very early 30s now), with my passive investments fully covering my cost of living at least in Mediterranean Europe (where I'm originally from) or South East Asia (where my partner is from), both amazing places to live :-)

Both my partner and I are not looking to have kids (we've thought about it for many years before reaching the conclusion) and we already have extended families with tons of kids, so from that point of view we feel there's nothing we're missing, it's a non-priority.

For everything else, I'm not exactly wasting my life right now, I'm enjoying it even in a very high cost of living area: if you can deal with housing (and not having kids admittedly helps a lot since you can live in way smaller and cheaper places close to work), things are not that much more expensive than anywhere else, so I get a lot of experiences with good arbitrage :-)

There are significant pro's. I become 10x the engineer working at FAANG in 6 months vs. the 2 years at my previous company. I worked on impactful projects with huge visibility, both internally and externally. Also, the started off "meh" but over the years, it's increased decently (mostly as a result of increasing stock comp).

I don't want to ruin all of FAANG for you.

The FAANG companies are not monolithic hiveminds. They are huge places where some parts are good and some are ... not so good. There are talented engineers and opportunities for growth at the good teams but you're not guaranteed to be in a good team at any given time.

I would say go for it but go in with a mission to understand the lay of the land and then transferring to a team that matches your growth goals.

Maybe make the move to a FAANG, just to get a real world idea of what that's like?

Your life journey is unlikely to be damaged by it, and the greater understanding of the world (etc) will likely be beneficial. Warts and all. ;)

Is there any chance having a chat to your husband along the lines of "Things at work really suck, I'd like to change by either really getting pregnant or changing job sooner rather than later. How're things with you?" is a good idea?

Burned out six months into my first post college job. Not as dramatic as some of the other failures here. Just stopped showing up, and refused to leave my room for a week. Eventually got up the nerve to call them and explain what happened. They were willing to take me back but I realized it'd likely happen again if I continued.

So I changed fields inside tech, from IT support/sysadmin to web development. On the job hunt right now so still not sure if it was the right choice. I have been enjoying learning it significantly more than the sysadmin stuff.

Once I scale up Azure analysis services(main data source for company Marketing and decision making) cause we were running low memory in peak hours, only to find out that I couldn't scale down, increasing cost by 300%. I had to recreate all the models again in the old tier increasing even more the cost cause we have to maintain both data sources till I finish with the creation of the models. My boss was mad.~azure lvl 100 destruction~

Here's my most memorable Microsoft snafu: My desktop was just upgraded to Windows 10. The "Favorites" links in File Explorer were converted into "Quick Access" links by the Microsoft migration process, but the aliases (titles) of the links had changed, or so I thought. I renamed the aliases back to what they were under Windows 8.

The phone started ringing with complaints, and to my horror I learned that I inadvertently renamed widely-used WAN and intranet folders. Quick-Access had no aliases: I had inadvertently changed the actual folder names. A colleague walked by and asked, "Why are you sweating and white like a ghost?" Fortunately I renamed them back before too much damage was done.

In my first job out of college, I had no training or experience in customer service. I was horrible at it without knowing what I was doing wrong.

For example, our system allowed for different ship-to's per order, but one vendor's receiving system wasn't able to handle multiple ship-to's. I should have referred them to my supervisor to negotiate something, but instead said, "you'll just have to change your system." They got irate. That and a few other "people skills" issues that needed tuning got me FIRED. I was devastated.

I had taken a "communications" course in college, but it was horrible in hindsight. The author and instructor should have been fired...uh, mentored, I mean. They should have made us read, "How to Win Friends and Influence People". Colleges have something against that book because it's not based on empirical data and doesn't try to sell the idea of using good logic. Instead it deals more with the "squishy" side of human nature. Humans are social creatures above all else.

Once worked for a company that had a subscription based service. They wanted a monthly report of expired accounts so that they could warm-call for renewals. The bug was that some accounts had paid for 2 years. After a couple years, someone noticed that the reports hadn't been including the 2-years subscriptions and it was effectively a loss of a couple million in revenue.

While I did code it to the specifications I was given, and was really clear on understanding it was still a pretty big failure.


Second biggest failure was checking in the db connection against a production database into source control. It was internal source control, and the transforms (ASP.Net Web.config transforms) had it anyway, so wasn't an increased risk in that regard. I was troubleshooting a production issue.

What happened is another developer had used that connection configuration to drop/recreate a table. It took a few hours to recover from (DB restore, then replay against API activity logs). And a day of lost time for a few hundred workers.

In the pre-HTML5 days of the Internet, I wanted to be a Flash Designer. Forever. I loved the platform and process so much, that I only had stars in my eyes for Flash! The idea of being able to blur the lines of animation, programming and design was wonderful. Then it died (April, 2010 https://www.apple.com/hotnews/thoughts-on-flash/) and so did my career.

Fortunately, in most failure stories we can find some ways to recover. Knowing Actionscript developed some programming skills, which were applicable to HTML5.

My key lesson is learning how to pivot from one technology platform to another. It's not always easy, requires some humility, but it's possible.

I developed an app many years ago , helping students find cheap eats around campus. The traction for people to use it was their but I didn't plan for the food stall owners to be extremely stubborn with prices and deals.

What happened?

0. Graduated from a top CS uni. Starting salaries were lower than KFC assistant manager. Started looking outside tech to for enough income to sustain being a programmer.

1. Put down $6000 on a coffee kiosk. Made real good money every day. But partners were all dishonest and manipulative.

2. Built an app for a guy. He paid well. Built another app for him. Contract sucked, one clause meant we couldn't get paid even though the project was 75% done.

3. Wedding planner business. Too much work, too little profit.

4. Brownies dropship, successfully distributed in petrol stations. Slow cash flow, dishonest suppliers.

5. Sold snacks and tarts. Did very well in festive season, but demand flatlined after.

6. Manufactured crab crackers. Unit economics didn't work out. Dishonest suppliers. Too capital intensive.

7. Did prototype for a big company. Contact person later paid me half of what he verbally proposed. Project had potential, but felt cheated.

8. Large tech project. Contact people were aggressive negotiators, trying to get insultingly unfair deals. One partner ragequit over it so we didn't pursue it.

9. Tried to do price comparison startup for groceries. Grocery suppliers didn't like it because it puts pressure on their low margins. Gov warned us that crowdsourcing is illegal if we didn't get grocers' consent.

10. Pivoted to recipes + e-commerce. Went remarkably well, but had to do some unethical things.

11. Tried to pivot to Blue Apron style service. Too capital intensive, not enough VCs here. Logistics and market were underdeveloped. (A YC startup ended up doing this reasonably well, Dahmakan, but with substantial changes)

* Success point! Gave up on the startup. Sold whatever I had of my startup at this point, at enough money to cover up for all the above losses. Bought an Alienware and 4K monitor, wasted 1-2 years of my life playing games.

12. Built app for a guy. Was paranoid of payment so there was a standoff where I refused to add features and fix bugs until client paid. Project was 90% done but client didn't close deal because of bugs.

13. Took on a hell project just to try to be a hero. Unreasonable budget, unreasonable specs, nobody wanted the job. Couldn't do it.

14. Worked on a startup for a guy. Too ambitious, architecture was too complex for team. Employees were overworked and unfocused, often "forgetting" to do things they were supposed to. They brought in consultants, managers, interns. We had 2 day sprints where half the time was spent in meetings discussing timeline.

15. Inherited company from late father as a non-executive director, hoping for passive income. A Fortune 50 company didn't pay, putting said company about $500k in debt/bankruptcy. Saw the red flags early, but hard to put foot down on family business.

16. Major project for multinational companies (plural). Top notch team. But VP insisted on unreasonable deadlines. 6 month project shortened to 6 weeks, at half the quoted price. Hacks were made, failed quality control. Teams blamed each other, started wasting each other's time to buy more time for their own work.

It seems like a lot but there's been some moments of success as well :)

Duuudeeeeee this was like watching a movie! You should totally turn this into a blog, perhaps even a book I bet you learnt lots along the way, you're literally the poster boy/girl for try, fall down, stand up, try Thanks for this :)

Yeah I can see how some people would get discouraged. But I grew up playing roguelikes. It was normal to try something, fail, then try again, understanding why one approach didnt work. It's actually fun to start something after a previous failure, because you'll be excited to try the path not taken.

If anything the mistakes I do regret is trying to settle for a comfy corporate job after selling my company. The failures were out of my control, and there was little learned from it.

That's really cool. I'd love to read each of those in more detail, if you have the time.

Yeah I suppose I should blog it. Never occurred that people would be interested in reading about failures.

A lot of the early stuff was me trying to help my wife with her business. Food made good cash flow but poor profit margins, and because of that the risk was quite high. We flipped to a more traditional role where she helps support my tech ventures now. Tech is high risk, but also much higher margins and thus less risky.

My freelancing career is actually going quite well despite all the setbacks here. The big thing was realizing that payment on a project basis is really bad.

Also it occurred to me when reading this that this is why people go to Silicon Valley. Every country has plenty of cheating, but it's worse in a bad economy, especially when you can get away with it. Dynamics of Silicon Valley make it harder to cheat/deceive someone and run away, so people can focus fully on building their businesses. The other half of failures is due to incompetent management, which is also something that happens much less in SV.

Me - a graduate student in a university. Looking around on all servers on the network and found a program called "crack". it was designed to reverse-engineer passwords given a unix passwd file. I thought, what the heck, let me see if I can get some gems. SO I used yppasswd to download all password hashes, and ran crack on it. Needless to say, sysadmin found out in a couple of days, and suspended my account for 2 weeks :(

Failure 1: Worked for a friend's company. As employee #1. I was to have all kinds of freedoms and responsibilities. Sounded really awesome.

Failure 2: Was too conflict-shy to draw firm boundaries, and likewise was he.

This friendship of 20 years is no more.

(While working for said friend) Failure 3: Offered a hair-raisingly badly thought-out fixed-price bid. Both my friends's mistake and mine. We probably sunk about 5x what we stood to make from it, and then abandoned the project, embarrassing a relative in the process.

There's plenty more of course, but these bother me.

Working remotely and contracting so I could travel the world. Went well until I burned out and ended up a broke alcoholic.

Me and my friend developed the a call center on the cloud software (contactcentral.io) .We feel product is on par with other competitors in the same space but we failed in marketing. We both came from engineering background bootstrapped ourselves, couldn't find better marketing strategy to position it properly by generating actual leads.Either we should have cash to spend on marketing or good minds working on marketing.

Published an app on the App Store that tried to connect to localhost instead of the production API server...

Back in uni, I wanted to be a motorcycle designer. I managed to get into Honda Japan's internship program, which was essentially a 3 day long interview process. They give you some design specs on the first day and then lock you in a room with 10 other kids to draw motorcycles non stop for three days. 4 years of effort all came down to these three days, and I couldn't draw anything on the first day. My mind went blank. I stayed up all night to catch up on the drawing. That was my first failure.

The lack of sleep meant I couldn't focus much during the second day. I pushed through and kept drawing. My next failure that night was to again forgo any sleep just to keep pushing out designs. On the third day, I was so exhausted by the lack of sleep, all the stress I piled on myself after all those years of expectations, My mind went numb during the morning's final prep. I was so tired, I didn't even know what the hell I was talking about during the final presentation, I interrupted the judges, I talked back, I argued. Naturally, I failed to progress to the next stage.

That failure was the beginning of a series of failures over the next several years.

I broke up with my fiancee to 'focus on my studies/career'.

I couldn't get a product/industrial design job because I was so focused on motorcycles during my studies that I was fairly terrible at most other products. Desperate for any job, I become a web designer for a company on the other side of the country. The job lasted six months.

I wasted money moving again to another city to search for better opportunities but found little. After another six months, I was out of savings and my family wouldn't lend me anymore money. I gave up, I contemplated suicide for a while, but finally moved back home to work for a relative.

About a year in, I got together with a couple of friends to set up our own design agency. One of them, talked the big talk but spent all his time playing games. Tried to rush work out the day before the deadline. The other worked really hard but wasn't particularly talented and decided to go back to his old job. I went back to work for my relative again.

After several years, found a small dev shop willing to take me in for not very much money. Spent two years there.

I'm in a better place now but those six years were terrible and whenever I look back at it, I've always felt that first failure at Honda just set a lot of things in motion. I was burnt out and depressed but I wouldn't admit it, so it just kept getting worst.

I have much gratitude for that relative who put up with me the whole time and gave me the opportunity, namely time to heal and a place to rest, to turn things around.

Failure #1: Joined a mobile app startup as employee #3 for a lot less money than I should have, run by a guy who really didn't know how to run a startup, he had just convinced the parent company to give him some money to make mobile apps. He couldn't raise money, only get handouts from the board. I falsely assumed because those people were rich as hell and the other companies they owned were successful that we'd be okay also. But they didn't want to spend hardly any money, especially on marketing, and when our apps failed to make money, it just convinced the board to pull the plug on us.

Which lead to Failure #2: The startup guy decided to make a couple apps as 'warmups' before his main app idea. When those apps were released and didn't make money, the board became more wary about letting us do the 'main' app. If we had just started with the main app to begin with, they wouldn't have seen those app failures and we could have probably convinced them to keep funding it until release and that app might have actually seen success. Having low-investment but visible failures made the company look like a failure, and insured its death.

But not a failure: I had a lot of autonomy, I had my first experience managing other employees, and the apps we worked on helped me learn a ton in a short amount of time, and I'm still proud of them years later, even if they didn't reach much of an audience. That experience also helped me land my next job.

I could probably do that about every job in my career, honestly. Pretty much every job I've had was a mistake and a valuable learning experience at the same time, at least up to a certain point. Before this job I had joined another startup as employee #3 that failed for different reasons, and after this job I joined a small company that ended up having its worse year ever and lost all of their clients, and then I took a contract job I despised just to keep afloat for awhile where the entire department fell apart when the lead dev left and they terminated my contract, but that gave me the skills that helped me get my current job I've been at for 3 and a half years which started so successful it was bought by a larger company, and has been very slowly eviscerated by that larger company as it has been forcibly integrated into the rest of its corporation (our company, now division, has about 15% of the workforce it had when it was purchased, and it had over 500 employees a the time), and yet I've learned a lot while I've been here and worked on projects for the most highest profile clients in my life (you can't get higher profile than them, really) and for the first time I've actually had to really, really care about optimization since our traffic is massive and constant, although the stress has gotten beyond ridiculous now and I need to leave soon.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact