The way I read it, you didn't destroy anything. You had a moment of panic, and could have created real problems.
Regardless, same lessons learned right? :) Whether the server is gone or not, it could have been, and the way we work changes drastically after a real world scare like this.
Did you ever tell the boss, or did you just let it ride? There's nothing like going to the boss in a cold sweat and the old "I f*ed up" look on your face.
I just let it ride at the time. When I joined the company there wasn't any other tech guy around except me (our tech guy was on a vacation). So I was left alone for a while, and had to sort out of lot of craziness :)
He put himself in a position where only chance protected him from a disaster. (Imagine if the update did trash the running server.) That's reason enough for a good sysadmin to be worried, IMO.
Never log into your production servers via shell? This seems a tad over the top.. especially if you are planning on writing a script to perform operations and need to discover dependencies / etc.
Well, the idea is that instead of logging into production, you log into your development environment, do the testing there--then you write a script that replicates what you just did. Lastly, you run the script on staging, and if that seems to work, then run it on production.
There's a pretty handy tool I use now (fabric: docs.fabfile.org) that makes it pretty simple.
How do you know that production environment and development environment are exact mirrors? I recognize you could image the production environment, but when setting up new applications I will make a variety of changes, especially during testing, which may require testing on the production server also.. especially since the production server is active and may not tolerate the changes the same way the development server will.
My point is, your method of "never logging into a production server" is extreme. You could have easily written a script for this and then by accident, executed it on the production server instead of on a testing server. Same problem manifests itself in a different manner.
"If you join a project with no version control, don't convince yourself that everything will be ok. Refuse to do any work until you version control it." Embargo on.
cd /tmp. Unzip some file with some system image - usr, etc, lib, the usual - that should be unpacked in /. Do what needs to be done. rm -fr /usr.
WAIT, NO! ^C.
"Backup, could you be so kind as to restore the ABC server /usr tree? I think I made a huge mistake. Thank you."
Good times. Learned a lot about why I need a VM with an image of a given system.
Regardless, same lessons learned right? :) Whether the server is gone or not, it could have been, and the way we work changes drastically after a real world scare like this.
Did you ever tell the boss, or did you just let it ride? There's nothing like going to the boss in a cold sweat and the old "I f*ed up" look on your face.