
The Day I Broke Production - twampss
http://lethain.com/entry/2010/feb/14/the-day-i-broke-production/
======
chuhnk
I'm sure this is no unfamiliar territory for anyone in dev or ops, especially
in a startup. My boss is what I like to regard as the master of all trades and
even with his 15+ years of experience has the odd hiccup in the production
environment. We are only human and that means we make mistakes. Experience
teaches you not to make those mistakes again. So for instance when I made a
config change in apache, synced it to all servers and restarted but forgot to
add a log directory, apache failed to startup and no requests were being
served. I wont make that mistake again lol.

------
swombat
Believe me, you'll be breaking production a lot of times again in the future.
Become comfortable with the thought of it - you need to be able to perform at
peak competence when production is broken, because that's when your skills and
clarity of thought are most needed.

