

All servers flooded by salt water - is it possible to recover data? - ivoflipse
http://serverfault.com/questions/447092/all-servers-flooded-by-salt-water-is-it-possible-to-recover-data-from-multi-pla

======
zackangelo
We had a flooded rack in New Orleans after Katrina. We had tape backups, but
during the evacuation frenzy the wrong tape was pulled from the library (we
should've just yanked the entire magazine). Not recovering the data would
result in thousands of lost man-hours of work, so when we discovered that the
tape we had was blank, it quickly became a do-anything-at-any-cost situation.

Things were dicey post-Katrina and they weren't letting anyone in the city. We
had to charter a sea plane and land it in a canal behind our office to gain
physical access. We were escorted by mercenaries armed with automatic weapons.
Once inside, we grabbed every drive and tape out of every server and array
enclosure.

We ended up being able to do a full recovery from tape once we got back.
Sometime later, when we were able to get the entire rack over to another data
center, I was curious if any of the gear would work so we reconstructed
everything. Miraculously, after being submerged for over 4 days, nearly every
hard drive and server booted with zero data loss (one or two drives didn't
spin up, but they were RAIDed). Perhaps even more impressive is that they
stayed running that way for years after in our staging and test environment.
It's pretty crazy how well-engineered and reliable the server-grade stuff can
be these days.

