
Unix recovery legend - araneae
http://www.ee.ryerson.ca/~elf/hack/recovery.html
======
shalmanese
I got into this situation once. I got pretty far into my recovery before being
stymied by the discovery that tar is only distributed via tar archives:
<http://www.gnu.org/software/tar/tar.html>

~~~
eru
Is a "shell archive" also a tar-archive?

~~~
jff
Nope, nor is a cpio archive, both of which are available for download at that
site.

------
jamesotron
I have experienced this very thing with a SunOS box at the remote end of a
satellite link. It was in the US and we were in NZ. Let's just say that you
can do a lot with bash and a single root shell.

~~~
jrockway
I would have hit Control-D just to get the free trip to New Zealand :)

~~~
epochwolf
You might earn a trip to the unemployment office instead.

------
blhack
I had a similar thing happen (having to recover from a very very stupid
mistake), I hadn't deleted everything, but I had moved the entire tree into
/var/www (oops)

This is an profanity-laden blog I wrote immediately after I figured out how to
fix it and got everything back up and running:

<http://newslily.com/blogs/94>

(I'll be the first to admit that I sound like an obnoxious teenager in this
blog, and that some of the things that I tried to fix it were _very_
naive...it was a few years ago that it happened and as stupid as it is to
admit, trying to get everything back up and running made my adrenaline pump
pretty hard, so...I apologize in advance...)

~~~
ajross
To be fair: you had this happen on a piece of consumer hardware which can boot
off of any of several dozen live CDs or USB drives, available off the internet
at megabit speeds using one of the (no doubt) several other pieces of consumer
hardware you have in arms reach.

The VAX 11/780 had no portable boot device, and BSD unix had no "installation
media". You bootstrapped to a system monitor, loaded a second stage off of
tape header into memory manually, and hand-build your root filesystem. And
that's assuming you still have your original tapes from Berkeley in storage
somewhere.

I actually tried doing this on the simh simulator a while back, out of sheer
curiosity. I gave up before managing to make it work.

~~~
blhack
_To be fair: you had this happen on a piece of consumer hardware which can
boot off of any of several dozen live CDs or USB drives, available off the
internet at megabit speeds using one of the (no doubt) several other pieces of
consumer hardware you have in arms reach._

I wish! This was done on a VPS located god knows where and run by a company
that was owned by people who didn't seem to speak any English. I remember
emailing them and asking them if they could just move my dirs back using a
chroot, but they had no clue what I was talking about. If I had allowed my SSH
session to time out, I would have most-certainly lost the server and
everything on it.

I have since switched to a combination of slicehost and linode.

~~~
cperciva
_... I would have most-certainly lost the server and everything on it. I have
since switched to a combination of slicehost and linode._

Switching to a different VPS provider is not the right way to prevent data
loss. Doing backups is the right way to prevent data loss. :-)

~~~
orborde
This is true, but he gets props in my book for being smart enough to switch
away. It seems surprisingly common that people don't do this.

------
FfejL
Single most important habit I've developed as a SysAdmin: measure twice, cut
once.

I never just

    
    
       sudo rm *

First, I do

    
    
       echo sudo rm *

inspect the output, and then use CLI editing to recall the command, and delete
the "echo"

I do this for damn near everything that involves a wildcard. It's a lifesaver.

~~~
rick_2047
I have added a like

alias rm='rm -i'

to my .bashrc. Sure its a pain when i have to delete whole directories but
saves my ass.

~~~
borism
this is default profile on almost any mainstream linux distro these days
anyway

~~~
rick_2047
Not on Arch

------
blasdel
More fun: libc is hosed on a remote box and you can't find any working
statically linked binaries with network support, but still have a few live
shells accessible.

I was starting to write a UUDECODE function in pure bash (I'd saved my ass
before by pasting uuencoded binaries into terminal emulators) — but then I
remembered that bash has built-in tcp support!

At least when called from the exec builtin to spawn a file descriptor, you can
open /dev/tcp/$host/$port psuedo-devices. It's existence would have saved me
countless more times, if the spoilsports at Debian/Ubuntu hadn't disabled it
at compile-time.

~~~
Kadin
> but then I remembered that bash has built-in tcp support!

You have to be careful making that assumption anymore, though; the version of
bash that's included with Debian (and downstream variants, e.g. Ubuntu) has
/dev/tcp and /dev/udp support disabled.

So it's not something you can count on having available in any random
environment you log into.

I disagree with Debian's decision to disable it, but they seem pretty firmly
set on it -- people have been arguing with them to reverse it for the better
part of a decade, to no avail, so I doubt it's going to change. It's a
declining minority of users who probably care. But you could do some neat
'party tricks' with it.

------
hassenben
This should only happen to you once in a lifetime.

and this is why my "rm" is replaced by this script... just moving stuff to
another directory (trash) and I clean my trash manually (with at least 2
confirmations)...

With all these precautions, if I delete something important, I don't deserve
admin rights :-)

randstring=`date "+%Y-%m-%d_%H-%M-%S"_bak`

while [ ! -z "$1" ]; do mv -b -S "$randstring" "$1" ~/.local/share/Trash/files
shift done

-hbt

------
stretchwithme
I wish there were an idiot mode for unix that would tell you exactly what the
command you just entered is going to do. I'd be turning it on all the time
when using commands I rarely use.

It would be nice to know I'm about to do something that I don't intend. As
long as the system doesn't ask me "are you sure you're sure you're sure?"

~~~
IgorPartola
Command line mysql client has a i-am-a-dummy mode where you cannot run a query
like "DELETE FROM _table_ ". You could of course run "DELETE FROM _table_
WHERE 1=1" but at least it saves you sometimes.

~~~
Davertron
This is awesome, I never knew about this; I usually type "delete from where"
and then fill out the query, just to avoid accidentally hitting enter after
typing "delete from users".

For me, this is sort of like standing at the edge of a big cliff; you just
have this weird feeling that you might accidentally, compulsively jump.

~~~
cynicalkane
Man, I thought I was the only one who felt like this (the cliff thing). I told
a friend once and he became worried that I was suicidal. (I'm not.)

~~~
epochwolf
I get that too :)

------
ErrantX
_If any of you ever have the same problem, here's the hex for future
reference:_

Bookmarked! :)

~~~
scdlbx
You work on a lot of VAXs? That's VAX assembler.

~~~
ErrantX
You never know!

(we actually have a huge pile of old old machines in a store room at the
office here - Im not sure if there is a VAX in there or not.. there could well
be)

I wasn't being too serious though :)

------
joeyh
My favorite recovery (though it was not fun at the time) involved a headless
machine that wasn't booting, and all I had available that would boot on the
machine was a floppy disk that ran a rescue shell. I modified the floppy to
start linux with console=/dev/fd0, and so was able to type blind and then
remove the floppy to read the output. After a few painful cycles of this, I
noticed I could often guess at what was happening by listening to the write
patterns of the floppy drive, and completed the rest of the recovery using
essentially morse code.

------
drhodes
I stopped $ rm -rf ./things -and instead do- $ mv ./things /tmp and just
forget about it. Yeah yeah, it won't work on a server, but for a home system
that reboots occasionally, it should work fine.

~~~
Pistos2
I've learned that nobody and nothing should use /tmp, on account of it being
777. The idea is that Mallory can set up a symlink from an anticipated
filename ("things" in your example) to an important file or dir. So writing to
/tmp/things destroys the symlinked file/dir.

------
mjhnghfh
Of course if they had been running the proper operating system with a proper
filesystem on the Vax then this wouldn't have happened.

You unix kiddies finally got a versioning filesystem now I hear ?

------
arethuza
Reminds me of my first job straight out of University (1988) - there was a
client who had a sysadmin who had a bad habit of logging in as root and
running rm *. This didn't cause any immediate problems - but all hell broke
loose when tried to reboot their system.

They had about 25 users on a 386-25 - quite a decent system in its day.

------
vorador
Does someone know what's this use of grep as cat he alludes at ?

~~~
olefoo
Match any non-empty line.

    
    
       /usr/bin/grep ".*" /path/of/file

~~~
nash
Sod any 'non-empty line'

    
    
      /usr/bin/grep '^' > output
    

Or even better, grep can be an editor with line filtering - if you make an
mistake, have it throw away the line:

    
    
      /usr/bin/grep -v '%' > output
    

(Any line with a %, throw away, keep the rest - good for multi-line files)

------
jurjenh
What if you were to symlink your entire tree to say /z - as far as I'm aware,
this would not use much more disk space, and as it would be the last to go,
could you not call /z/bin/whatever ?

I may be way wrong here, but I thought the symlink would still work with the
original being deleted, because the file system won't actually mark the file
as gone until all links are gone.

Would (or does) this work? Haven't got access to a unix box right now, so
can't try myself, but thought it might be a naive failsafe (at least until you
actually wait for _rm -rf_ * to finish...)

~~~
yason
Symlinks won't save you but you could use hardlinks to make two pointers to
the same file. You would have to do it for each file — was it _cp -l_ which
would do that.

~~~
pronoiac
On Linux, I've used "cp -al" to make a forest of hard links, consuming little
space.

------
zellux

      # cd 
      # rm -rf *
    

I'm wondering why they can delete files under /bin and /dev?

~~~
rdtsc
Because they are root and those are just files.

Perhaps they need to alias rm to 'rm -I' (-I : prompt once before removal)? I
remember one of the MandrakeLinux versions had that enabled by default many
years ago. It saved me a couple of times when I was still a Unix noobie.

~~~
zellux
AFAIK a single _cd_ command will set current working directory to /root, and
_rm -rf_ will only remove everything under /root. Or does _cd_ command had
different behavior in Unix at that time, comparing to shells today?

~~~
olefoo
That sort of thing is why /root is now root's home directory.

------
pfarrell
Awesome story. Thanks for the repost. I've never read this one.

------
barrkel
I appreciate the filesystem snapshots that ZFS give me.

~~~
donw
Fairly certain that zpool and friend live in /bin or /sbin, so you'd still be
pretty hosed if you lost all your system binaries.

~~~
barrkel
The choice of what snapshot to mount is part of the boot process, as chosen by
grub. Nexenta's apt-clone creates a new snapshot for every system upgrade and
adds a new boot entry for rolling back in case it doesn't work.

------
xtacy
if you could recover "cat" and if there was /proc.. oh well, lots of "if there
were".

~~~
rdtsc
Perhaps just use shells with more builtin commands? Something like busybox. So
if you ever do rm -rf * and don't close your shell you still have all the
basic utilities available.

~~~
jedbrown
Busybox is a binary that inspects argv[0] to determine how to behave. This is
very different from shell builtins that do not launch a new executable, and
will thus function correctly after the executable is remove from the file
system. As far as I know, you cannot get busybox to perform it's other
functions as "builtins". But if you have a functional debugger, it should be
possible, though computing offsets is likely to be hard without another
machine (chances are that the busybox binary has been stripped).

~~~
pronoiac
While busybox is running, if /proc is still there, you could probably use its
file descriptor to get it back.

------
gcb
while a newbe i asked on irc what would happen if i move /usr in my sgi indy
to a new scsi drive i just installed. sice no one replied i opened a terminal
and `sudo mv /usr /newusr; sudo ln -sf /newusr /usr` or something dumb like
that.

While i waited i opened netscape and went to read something. Then while it was
coping irc was flooded with bad omens and curses about what would happen if i
did that on a live system.

The only thing i noticed was that the icon state (the desktop shortcuts
changed picture if there was any process open for their target files) would
some time blink. But that was it.

Not a thing happened. Even the bookmarks i saved on the netscape bookmark file
opened before the move ended up on the right place.

