

The Time I Accidentally Destroyed a Production Server - b14ck
http://projectb14ck.org/how-to-not-be-lean

======
dpcan
The way I read it, you didn't destroy anything. You had a moment of panic, and
could have created real problems.

Regardless, same lessons learned right? :) Whether the server is gone or not,
it could have been, and the way we work changes drastically after a real world
scare like this.

Did you ever tell the boss, or did you just let it ride? There's nothing like
going to the boss in a cold sweat and the old "I f __*ed up" look on your
face.

~~~
b14ck
I just let it ride at the time. When I joined the company there wasn't any
other tech guy around except me (our tech guy was on a vacation). So I was
left alone for a while, and had to sort out of lot of craziness :)

------
bschlinker
Never log into your production servers via shell? This seems a tad over the
top.. especially if you are planning on writing a script to perform operations
and need to discover dependencies / etc.

~~~
b14ck
Well, the idea is that instead of logging into production, you log into your
development environment, do the testing there--then you write a script that
replicates what you just did. Lastly, you run the script on staging, and if
that seems to work, then run it on production.

There's a pretty handy tool I use now (fabric: docs.fabfile.org) that makes it
pretty simple.

~~~
bschlinker
How do you know that production environment and development environment are
exact mirrors? I recognize you could image the production environment, but
when setting up new applications I will make a variety of changes, especially
during testing, which may require testing on the production server also..
especially since the production server is active and may not tolerate the
changes the same way the development server will.

My point is, your method of "never logging into a production server" is
extreme. You could have easily written a script for this and then by accident,
executed it on the production server instead of on a testing server. Same
problem manifests itself in a different manner.

------
armored
"If you join a project with no version control, don't convince yourself that
everything will be ok. Refuse to do any work until you version control it."
Embargo on.

------
trbecker
cd /tmp. Unzip some file with some system image - usr, etc, lib, the usual -
that should be unpacked in /. Do what needs to be done. rm -fr /usr. WAIT, NO!
^C. "Backup, could you be so kind as to restore the ABC server /usr tree? I
think I made a huge mistake. Thank you."

Good times. Learned a lot about why I need a VM with an image of a given
system.

