
Why prod is down - napolux
https://www.gruntwork.io/why-prod-is-down/
======
sdfjkl
The underlined placeholders aren't inputs and cannot be filled out.

Also missing are a few options on what to do, including: "Let it burn", "Throw
more gasoline at it", and "Quit while you still have some sanity left".

------
gozur88
>Worst of all, our database is not backed up, so we lost our data backed up,
but the backups don't work, so we lost our data sharded, so we lost our data
twice NoSQL, so we're webscale, and we've been losing data for years

Heh. What I keep running into at various companies: "Backed up, but for some
reason the restore is going to take three days (if it works), and it will be
faster to tap original sources.

Why are we doing backups, then? Company policy.

~~~
itsjustjoe
Because backing up stuff is super easy, but no one actually tests restoring
data until they need to do it live.

~~~
apple4ever
We'll do it live! We'll write it and we'll do it live!

------
davidspiess
It's scary how relatable it is

------
fhood
My goodness, this is a work of art.

------
MarkMMullin
We failed over but there was nothing there so we can't fail back -- eek, we're
dead :-)

------
agotterer
Somewhat related, has anyone used gruntworks? Thoughts?

------
greatNespresso
Is that real ?

~~~
dsr_
Yes, those are real words written by a real person, stored on a real server
and delivered to you by a real Internet.

A better question might be: "Is this really a copy of a letter written by a
particular person to another particular person describing a particular
incident?" to which the answer is "No."

An even better question might be: "Do the problems outlined in this document
really occur?" to which the answer is "Yes, frequently."

