
When Things Go Wrong - abuggia
http://eng.localytics.com/when-things-go-wrong/
======
_Codemonkeyism
I disagree with

"Why didn't [Engineer X] know enough SQL to understand that query would break
the database?"

being bad (While "We need to shut off his/her access." is).

Post mortems have many different levels, technology -> people -> processes ->
org -> culture. It's not wrong talking about people, it's wrong to blame them
though and stop with people. Asking 'Why?' only stops with company culture.

Usually people make mistakes, they don't want to break things. It doesn't help
to not talk about people, but it's important to ask why the person in question
acted the way (s)he did. Would others do the same? (most probably) Why? UI
issues? Documentation issues? Too many concurrent warnings? etc.

Also important to look from the situation forward (incident happening), not
from now backwards (hindsight).

To the SQL question one could ask about the training, reviews, ...

~~~
tonyjw
Author here. I agree with you that training engineers on SQL would be a great
action item coming from a postmortem. I was trying to emphasize resisting
shutting off access as "the fix" for the problem. We don't want to blame
engineers for not having the knowledge - we want to level up their skills if
that contributed to the issue. I think you're asking the right kinds of
questions.

------
Down_n_Out
Is anyone else getting this (using uBlock with FF):

    
    
      uBlock Origin has prevented the following page from loading:
      http://eng.localytics.com/when-things-go-wrong/
      Because of the following filter
      ||localytics.com^
      Found in: Malvertising filter list by Disconnect • Basic tracking list by Disconnect

~~~
jlarocco
Yes, and after Googling what they do, it makes sense.

~~~
Kiro
Blocking the actual page as well and not just scripts? Seems broken to me.

~~~
jlarocco
Not really. Why waste time blocking specific scripts when the whole point of
the company is to be sleazy?

There's really never a situation where I want to download something from a
site like that, because I don't trust them at all and I completely disagree
with what they are doing.

It's not worth whitelisting - I can read something else.

~~~
Kiro
Good for you. Dealbreaker for me.

~~~
bdarfler
Thanks to gorhill there is now an update to uBlock to address this.

[https://twitter.com/gorhill/status/776183271210229760](https://twitter.com/gorhill/status/776183271210229760)

------
dudul
The whole blameless postmortem is kind of laughable coming from a company well
known (at least in the Boston area) for a culture of abuse towards engineers
and a high turnover.

~~~
cantstopthis
Huh? Abuse? I've been working in the Boston tech community for a long time and
have not heard this. Localytics as an engineering culture has always been top
notch. Yes, the company has lost people, good people at that, but the team is
still doing very innovative things. Ive spoken with people who are there now
who still say its still a very good engineering environment.

------
kevan
I'm a big fan of using chat as the real-time incident log even when we're all
in the same room. It's trivial to generate a timeline of events for the
postmortem and there's a lot of info you can extract from the log during the
postmortem process.

~~~
Kiro
Yes, I think that's what has added most value for us after introducing Slack.
It's extremely useful.

~~~
newman8r
I'm working on some communications software myself - so one of my hobbies is
writing down as many UX issues as I can find with slack/hipchat/etc etc

Have you noticed anything that you'd change? I actually like slack for the
most part, but I still find some UX smells in it.

And I actually use it for incident logging too.. never really thought about it
but it's one of my favorite uses for it - and it helps the client on that job
see the response times and get a general overview.

------
homero
I wish there was real world analytics

~~~
Kiro
What do you mean?

~~~
homero
Like those things that beep when you walk into the store but more silent ones
making a heat map of your whole store with other analytics

