Corporate accounts payable, Nina speaking. Just a moment!
...for 10 hours: https://youtu.be/oFwkL1WgD_0
Hopefully at least two people reviewed the commit before it was put into production. Someone set up the post-launch smoke-tests/api-test-suits. Someone built the CI/CD-system. Etc etc.
My point is, it is rarely a single individuals fault when something at this scale goes down.
I could not agree more with this statement.
When discussing success to an outside group (giving a presentation or something) or higher up (especially here!) you always use "we". Even if it is a section that only you worked on or you did most of the work. You can often pick out good managers by just seeing if they talk like this, if their focus is around what they individually did or what their team accomplished.
My point is, it is rarely a single individual's success when things work.
There's a balance and I do think we tend to focus on the individual as a society. We can only climb mountains by standing on the shoulders of giants. One person gets to the peak and we should praise them for that, but we shouldn't ignore their foundation either (which I think currently happens). E.G. Kepler was an essential part of science and needs to be recognized, but if it wasn't for the work that Tycho Brahe did, Kepler's achievements couldn't exist. The "single person" mentality ignores the importance of the foundational work that needed to be done and frequently causes many to feel that they are not achieving simply because they are working in these roles, which are essential.
But the parent is right that it still sucks to be the persons whose work was the proximate cause of an outage, even if nobody is going to blame/punish you for it. It doesn't make for a day where you feel good when you get home from work.
This is a form of illusion. A cover up for what is essentially in reality a mistake made not only by the team but by the individual as well. To mask part of the truth and lay only the blame on the team is an effective form of not hurting someones feelings but also an effective form of avoiding the full reality.
Do people not see how illogical it is to blame the team but only praise individuals? The precedence being set here is that: People succeed as individuals but fail as teams.
The harder ideal to strive for is that both the team and the individual take the blame, but it's harder because people are prideful stubborn and easily hurt.
If you make a mistake step up and admit your mistake. Don't hide in the corner and expect the team to change all their processes to account for your mistake. Yes the team should do this, but yes you should stand up as an individual and do things yourself as well.
There are times when you must blame an individual. Let's say a team member repeatedly pushes bad and buggy code to production. Is it the teams fault or the individuals fault when such actions are repeated? Does it lay on the team or the team member to make sure buggy/bad code doesn't go into production?
The line is blurry here and I feel it is ultimately the wrong stance to say absolutely good teams don't blame individuals. Good teams and good people take responsibility so there is no need to dish out blame.
What happens when the person who spilled coffee all over the on premise servers doesn't take the blame? The team sees this and decides to blame nobody. Is this a good team? No.
The good team member volunteers and states publicly that the fault is his own. The good team agrees with this stance and also says that the fault is with the team as well and both the individual and the team take steps so that this mistake cannot happen again.
It isn't about eliminating recurring problems, it's having the highest chance of reduction. If someone continually performs poorly then thats management's responsibility to replace them.
If your system can't tolerate one person's mistake, it's not robust.
None of this means you can't praise a whole team (you should! The original statement just said you CAN praise individuals), or discipline team members who regularly show poor judgement.
Isn't this contradictory with the original premise? If you discipline an individual you are blaming the individual.
You're saying you can praise individuals and you can discipline individuals and you must also follow this: "Good teams don't blame individuals. You can praise individuals, but you take blame as a team."
Do you not see the contradiction? I'm literally getting voted down because of slightly miffed feelings.
Despite all of this the logical correctness of my statement stands and you have inadvertently agreed with every aspect of my statement.
If I'm continually pushing buggy code, that's something that I'm going to be personally held accountable for. Blamed if you will. That's because I'm the only one involved. My team isn't constantly writing buggy code, I am.
If I push buggy code into production and take down the site then the team is to blame. Why is our process setup such that I can do so. EVERYONE, even the best programmers will make mistakes, and if the process allows those mistakes through to production then the TEAM has failed to build a suitably robust process, not the person who happened to write a bug, because every human being will be that person one day.
Your taking a once sentence philosophy about how to deal with technical incidents and treating it like we're advocating it as a universal and moral absolute. No one is claiming that if I throw a rock through someone's windshield my whole team should be arrested. What we are saying is that is I make a slip up and write a bug that takes down DDG, I shouldn't be held accountable for taking down DDG. If there's a larger pattern of poor decision making I'll be help accountable for THAT, but it's completely 100% separate from the outcome of knocking production offline.
I think a lot of vote downs are emotional and reactionary rather than genuine disagreement. I think that's the majority. I can't fault people for this as it's part of human nature but I don't think this type of voting is ever officially sanctioned by HN.
>If I push buggy code into production and take down the site then the team is to blame.
And you are not? I propose that you and the team should take the blame. People make mistakes but all the time and they shouldn't be singled out for it, but I don't think it's good etiquette for the team member who made the mistake not to own up to the mistake either.
>Your taking a once sentence philosophy about how to deal with technical incidents and treating it like we're advocating it as a universal and moral absolute
It's a one sentence philosophy that doesn't hold any logic in my opinion. I'm not saying anything is morally absolute here, I'm trying to say that the quotation doesn't even hold up in even the most basic situation.
It's elegantly written, no doubt, but I think that elegance is deceptive and that's why I wanted to say something. The real world in my opinion is nowhere even remotely close to that philosophy. I would argue that philosophy is just a surface level special case. In the real world people get blamed and fired all the time.
Also, just a bit of friendly Internet advice, this isn't the best way to interface with people:
> Despite all of this the logical correctness of my statement stands and you have inadvertently agreed with every aspect of my statement.
Well I'm getting massive vote downs and the original poster literally called what I'm stating as initializing stupid blame games to win the stupid prize. Kind of rude and insulting. I think it's reasonable to say that my reaction is reasonable given the attack.
It's not exactly the real world on HN either as people are harsher than normal, ruder and less receptive of differing opinions.
I would say that Giving people a piece of "friendly advice on interfacing with people" won't exactly win you any friends either. If you teach people things using that methodology you will get a retaliatory response whether it's on the internet or the real world as the tone is a bit domineering and elitist.
>Blame is for incidents. Discipline is for patterns.
This doesn't make sense to me. I can't blame people for patterns? I can't discipline people for incidents?
The sort of short and terse elegant expressiveness of the phrase ultimately hides a statement that makes no logical sense. Typically someone is blamed first before he is disciplined. Both go hand in hand.
Either way, you're probably just trying to say that let the team get blamed but if you see patterns then blame the individual. Not as elegant but no linguistic tricks that decorate ultimately illogical statements to appear as philosophies of life.
I never said play the blame game. I never said the team must not take the blame. Please don't put words into my mouth.
I literally said it's the teams fault and the individuals fault. Both parties must take responsibility.
Take it this way, if a single team member repeatedly makes mistakes then is it the teams job to put methods in place just for that team member? Or is it the teams job to help that team member as an individual?
The blame game is a stupid game so don't play a blame game. Take responsibility both as a team AND as an individual. The team taking the blame exclusively is AVOIDING individual responsibility. It is not a sign of a healthy team.
I am essentially saying the solution is not so clear cut. The team can't always take the blame just to "avoid a blame game." The reality of the universe is that problems aren't always team level problems, that problems at the individual level exist as well.
To exclusively avoid addressing problems at the individual level is a stupid and delusional endeavor and you also win the stupid prize for ignoring reality.
>If someone continually performs poorly then thats management's responsibility to replace them.
This is called blaming an individual then firing him. It entirely contradicts your main argument. I didn't even go there yet, I advocated blaming the individual than helping him improve as a team. You immediately cut his head off and let management do the dirty work.
It perhaps is related to devops or the search index cache servers is broken
* You probably got a review on your change before it was submitted, so you weren't the only one to think it was an ok change to make
* If you "revert first, ask questions later", you might also be the first one to fix it
* Hopefully the team recognizes that there are higher-order causes that caused your submit to break things than your own personal skill
EDIT: scratch that. DDG is back in business again. I would love to read a post-mortem.
"These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google)."
Edit: strike that, only for some entries. Must be some cache?
I was quite confused why my query for a well-known brand name was returning nothing. Bang queries still work.
- It could be an issue with DNS servers
- Certificates that got expired
- Latest push to production that got wrong
- DDoS attack
[EDIT] removed the metrics due to downvotes! check below for traffic information and how much is lost..
"Google suffered an outage late on Friday night, though you might not have noticed -- it only went down for five minutes. During that brief window, Internet traffic around the world dropped by a massive 40 per cent, according to Web analytics firm GoSquared, Sky News Online reports. "
Bing also: https://www.pcworld.com/article/183770/bings_30_minute_outag...
"Microsoft's Bing search engine was offline for about 30 minutes on Thursday evening."
"Qwant’s infrastructure broke down around 12.00 UTC, and the service could only be restablished hours later, around midnight"
Edit, and now it's back up for me, so ~15-20min of downtime from my perspective