Hacker News new | past | comments | ask | show | jobs | submit login
Do big companies like Google and Amazon fire developers for buggy releases? (quora.com)
37 points by rms_returns 30 days ago | hide | past | web | favorite | 22 comments

Not going to answer for Google, but my company, which I assume has a philosophy pretty similar to Google, will typically not fire developers for buggy releases, but not for the crappy reasons mentioned in the OP.

The general idea is that:

- everybody writes buggy code, once in a while – living under the pressure of being fired for an error that anybody could have made is not conductive to innovation and experiments;

- if you have released a piece of software with a bug, you're now expected to fix it or help fix it and make sure that it never happens again to you or anyone else – not run away and update your resume;

- both learning about the bug and fixing it are learning experiences – at this stage, the company has already (involuntarily) paid for you to get this experience, so fire the person who is now the most knowledgeable about a problem?


We accepted that bugs are fact of life, and instead of burdening the engineers with fear, we reduce the scope of damage through data: we use canaries, experiments and monitoring to limit the damage and react fast.

Going against human nature as a result just slow things down. The other benefits to above setup is engineers are less afraid to make mistakes and are willing to try ideas more since it is now very cheap to try them

There are times when you need to be more thorough such as medical devices, space programs, and so on as they might risk human life, but for the most part you will go really far with above

Disclaimer: current google employee

In 30 years of software development, I've never seen a dev fired because of a buggy release. You get fired for 3 reasons: long, ongoing performance problems; violating company practices like stealing or sexual harassment; or when the company needs to cut salary and staff to meet numbers.

Also, I've never seen a developer who behaved like S did in the story.

I'm always wary of stories such as this one. Everytime I encountered people complaining they were doing everything by the book while that one guy/gal over there gets all the credit doing mediocre work, there were more variables involved. Maybe H is a close friend and S has a bitchy resting face the author can't get past, or maybe H takes so long polishing his/her code and understanding business requirements that management never trust him/her with bigger tasks, while S gets the proverbial thing done fast and good enough for the business meeting next week. Don't get me wrong, I love me some beautiful and tested code that works well, but it takes time, and sometimes time is imperative when you care about money, and in a business setting, that's always.

I have never seen any case of a developer holding the project to ransom in order to fix a bug, as alluded to in the article.

That said the key point that those who are always firefighting and ‘saving the day’ do often get the recognition as opposed to the people who squat the bugs early, follow sane processes and test their own stuff well is true.

But the question is about tech companies (not outsourcing shops like in the top response I saw). And I think the answer is a simple no.

This is so funny. I once worked on a year long project. The aim was to reduce customer escalations for my client. Changed the architecture of an extremely complex software making sure no regressions occurred and all new features worked flawlessly. The software was released and no customer escalations. I was happy, but found no one else cared. A few months later, during a meeting with the client manager, I brought this point up. I was told that since I delivered an error-free software, people forgot about it and that is a good thing. I wad told that if I wanted to get noticed, I should firefight instead of writing bullet-proof software.

This is similar to all the fixed issues we solved for Y2K. When nothing happened then rather than saying 'well done to all who fixed the issues' it became 'that was a waste of time and overblown as nothing happened.' Sometimes I think they wished a plane fell from the sky due to a software bug so they could slam software developers for incompetence.

There's no middle ground :(

Planes fell and software was blamed. By extension, the developers were blamed too.

My take is that you don't get rewarded for it directly, but rather once you're finished, you can move on to the next thing without having to constantly firefight and squash bugs on the previous project. So ideally you can get a lot more done in a given half than someone who pushed out a pile of buggy spaghetti that they're now constantly having to deal with.

On the other hand, if you can ship something a few weeks faster that might have a few bugs, but the customer never notices, I can understand why a business would reward that.

True, but my experience is that managers are not so great at recognizing consistent good results but often remember someone coming through in a clutch situation. Because measuring developer performance is hard, the consistently good dev is often overshadowed by the one who put out fires (even if they were the firestarter).

I agree with you.

I seriously doubt any successful software company is firing engineers for introducing bugs. They'd have no employees.

> One Google engineer from back then says the most remarkable thing about the co-founders' code was that when it broke, users would see funny error message: "Whoa, horsey!"

Right. The developer made the bug, perhaps QA should have found the bug, perhaps the project manager has the responsibility. There is no single person to point out.

That said, it is not normal, in my country at least, to fire people.

I've worked at Google and Amazon and they would never fire a developer for a buggy release. All engineers write buggy code. With mentorship, code reviews, and reliable test infrastructure, they write fewer bugs. With strong QA and monitoring they ship a lot fewer bugs.

Google and Amazon are not dumb enough to fire an engineer who would cost 10s to 100s of thousands of dollars to replace because that engineer happened to get unlucky and push a release that slipped through all those cracks. (Doing so maliciously would be another story, and would warrant firing.)

Basically the vibe I get after a few years on HN

Small company : fired. Assume someone out there is better than you. Unethical due to desperation. Probably why company remains small unsuccessful.

Medium: Put in blacklist and harassed with reviews until you repair your credibility or leave voluntarily.

Large company: Can’t fire you as that would affect their reputation. Alienation within company until you repair credibility or leave.

The large company description is speculation on my part and could be wrong.

What is with the irrelevant stock photography throughout the post?

It's Quora. They pretend to discourage answers with such a low weight-to-usefulness ratio, but in reality those are highly rewarded - not only by the users with upvotes, but by Quora themselves by promoting those answers even in excess of their upvote count and awarding the authors Top Writer status etc. It's the way things are done there, and it's why (despite having been a Top Writer myself for two years in a row) I left.

At traditional product-oriented companies, buggy releases are recognized as a programmer-performance issue and dealt with eventually (though as others have pointed out immediate firing would be absurd). At modern internet-facing companies, developers who produce buggy releases are rewarded. It's not intentional. It's what happens when performance is evaluated 100% on releases regardless of bugginess. If you push lots of code, you get rewarded. If you slow down to ensure quality, or spend time fixing code that already exists, you get penalized.

BTW my impression is that, within that peer group, both Google and Amazon are on the better side. They don't have "move fast and break stuff" as a core engineering value.

The company I work at does not fire people for introducing bugs (everyone introduces bugs at some point and time). We try to find out what failures in our process (dev, test, release) contributed to the failure and try fix them. Some of the causes may be large, so it takes us time to get the organizational commitment and focus to work on them, but we do act.

I have (personally) been fortunate never to have worked with an “S”, so I think the author is being overbroad.

I noticed that the author called out several contracting firms instead of "real" tech companies. Can't help but wonder if things really are that messed up on that side of the fence.

Everybody writes buggy code and those who say they don't are kidding themselves. The most important thing is how you react to bugs being found. The right answer is to acknowledge the problem and fix it. Write a unit test so it never occurs again in this code. If you get hit by a common type of bug that's preventable in code, often, then your unit testing is not up to scratch.

If you write software for production and don't do unit testing then you are in the wrong job.

Haven’t seen anyone fired in a year at my current large company. Have seen our process deployed many times to make sure work is coordinated to make that error impossible next time.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact