

Ask HN: how do you backup your servers? - adnam

I need to think about backing up data I have on a server, and would like to know how <i>you</i> do it. Do you backup whole machines or just some specific directories? Do you do off-site backups, or local? How much data do you have in a backup job? What's you preferred method: tar, sftp, scp, rsync, paid-for services etc? What's the most annoying thing you encounter managing backups?
======
hga
I divide my backups into three types:

    
    
      Bare metal.
    
      Frequently changing data.
    
      Occasionally changing data.
    

Everything is weighted by how automated I can make it (e.g. everything but
some day to day monitoring), what worse case recoveries are possible (no one
cares about backups, only recovery) and time to recovery (my personal use, so
a few hours are OK). And the threat level, e.g. a fire in one room
(sprinklers) and the possibility a tornado would take out the local hardware
but leave me alive (I live adjacent to tornado alley and have lived through a
couple).

So I have a bare metal backup of a compressed image of my system drive (made
with dd using a live-CD) which is updated every time I mess with LVM and
nightly snapshots of my root partition (LVM snapshop of partition with dd).

From that toe-hold I can rebuild my system from any worse case short of
hardware destruction, I use nightly backups with BackupPC for normal system
recovery, augmented with what's below for frequently changed data.

Frequently changing data, like email and code I'm working on, is rdiff-backup
snapshotted every hour. If I was maintaining customer data I'd do a continuous
process.

The same data is rsyncsrypto-ed and most of that is rynced to rsync.net.

Occasionally changing data is rsynced every night.

I try to avoid common mode failures, e.g. no one piece of software, one system
or one disk will cause total data loss.

Ask if you want particular details; for a long treatment of this, read _Backup
& Recovery_, " _Inexpensive Backup Solutions for Open Systems_ " by W. Curtis
Preston ([http://www.amazon.com/Backup-Recovery-Inexpensive-
Solutions-...](http://www.amazon.com/Backup-Recovery-Inexpensive-Solutions-
Systems/dp/0596102461/)). It's getting to be a bit old, but a lot of it still
holds and the principles will be true for a long time.

(Note, while the above particularly paranoid approach is for my personal
systems, it's based on harsh experience going back to 1977 (sic), including
plenty of small scale "enterprise" systems where total data loss was
unacceptable and a couple of 24x7 Internet startups.)

------
Mc_Big_G
Cron + S3 + s3cmd + small custom script for DB dumps

Cron + snapshots + small custom script for EBS on EC2.

Basically, I store all non code files on EBS and redundantly backup to S3,
which amounts to living or dying by Amazon's hand.

Recovery consists of firing up a new EC2 instance, attaching a snaphot and
running a deploy script, which is handled automatically by a different EC2
instance, assuming I haven't lost all instances. Manual recovery in that case
would be time consuming, but doable.

Snapshots are nearly instantaneous and uploads to S3 from an EC2 instance are
about as fast as you can get.

Most annoying thing? Testing recovery.

~~~
hga
" _Most annoying thing? Testing recovery._ "

Indeed ... but at least you do that!

One of the great things about clouds like AWS is that you can now do an almost
complete bare metal restore test (modulo stuff like your official DNS vs.
test) for a small amount of money instead of trying to justify buying a
separate system for that sort of thing (which in the real world I was never
able to do, although I fortunately never suffered a total data loss despite a
few close calls).

------
scottw
I backup entire disks using a towers of hanoi algorithm with (wait for it...)
good old dump(8) to a second local hard drive. This method keeps 7 full days
of backups (using an incremental diff, so it's fairly space efficient). I've
used this method for almost 15 years and have used it dozens of times to
recover missing things. dump has never failed me!

On my other servers I use rsync just to keep a current mirror around. I always
ask myself, "How hard would this be to recover if I lost it?" and "Will I need
to go back in time for some things?" Depending on the answer to those
questions I make my backup strategy for the server in question.

Yesterday's Daring Fireball post on the same topic said it best: "Every hard
drive in the world will eventually fail. Assume that yours are all on the cusp
of failure at all times. It’s good to be spooked about how long your hard
drives will last."

~~~
cnvogel
I've used xfsdump/xfsrestore on Linux and xfs-filesystems (obviously) in the
past. Very nice stuff, I second the recommendation.

Of course it only provides for the raw data-transfer task so one has to think
about the scheduling and planning yourself.

------
brk
Generally rsync to a mirror-image hardware box in the same location.
Everything is backed up so any fault/failure on the primary machine can be
recovered (in theory) via the hot-standby machine.

Critical user data, databases, etc. are dump'd/tar'd/gzip'd to a remote
location once a day, generally held as long as space permits.

------
ErrantX
Considering the work we do (corporate and criminal computer forensics) backup
strategies are about our second most important thing :D So naturally it gets
neglected...

Gold Images: of the various OS configurations and requisite software so you
can redeploy a server fairly quickly

Diff Backups taken nightly of our various data stores / databases

Weekly backups of main server data (MySQL, MSSQL, Hadoop clusters etc)

In terms of software we use a couple of commercial solutions plus Bacula
(which isn't really that great unless your using tapes...).

Inspired by Tarsnap (the encryption and so forth) Im in the process of
speccing up an idea to write out own in house version (I'd love to use Tarsnap
directly but not a chance unfortunately) to replace our mish mash of software.

~~~
gommm
Why not a chance to use Tarsnap? just curious...

~~~
ErrantX
We cant transfer data offsite (even encrypted) for contractual and legal
reasons (or more succinctly: I'd be locked up if anyone caught us doing that
:D)

------
lrm242
tarsnap + bash script! <http://www.tarsnap.com/>

Tar-like interface, data deduplication, awesome support, fair pricing. What
more could you ask for?

~~~
cperciva
_What more could you ask for?_

You forgot security. :-)

~~~
lrm242
Security is implied because I know you programmed it :).

------
avar
I'm just a mortal developer so I don't have to admin anything serious, but
here's how I do backups of my personal files & server. Mainly using Git:

Any code / changes to Debian's default /etc setup go into Git and are at least
on both the relevant machine (laptop or server) and GitHub. See
<http://github.com/avar> and <http://github.com/avar/linode-etc> for the /etc
on the server. I keep some personal data (like finance notes) in GPG encrypted
files in private Git repositories.

E-Mail is backed up my Google's GMail (presumably) and data like my small
collection of music is something I can always get back from the Internet.

If the filesystem on my laptop, server or GitHub is destroyed right now I can
get everything working again relatively quickly just by installing Ubuntu or
Debian from scratch and copying my home directory, /etc and code from Git on
one of the remaining machines.

I've made a script that automatically backs up my Git repositories daily
(<http://github.com/avar/github-backup>), the resulting backup is around
180MB. Losing some of my other data would be an annoyance but this is all I
/really/ care about.

Follow somebody else's advice if you need "real" backups, but for my simple
needs this suffices.

------
dunhamda
We've been using Bacula: weekly fulls and daily incrementals to disk, and
weekly fulls to tape for offsite storage. As our data size grows, tapes are
going to run out of steam. I've recently began experimenting with
<http://www.nongnu.org/duplicity> to do encrypted backups to S3. So far it has
been working great.

------
bryanh
We do tar'ed directories/databases onto S3 and this script:
<http://simonwillison.net/static/2008/backup_to_s3.py.txt>

Right now its only done weekly, as we're not experiencing high growth. We also
store downloads, so the tars can get very big, but it works well.

------
fnid2
I backup code to flash once a day in case of hardware failure. Once a week I
swap flash drives into a safety deposit box. I also backup any local databases
to flash once a day. Use a scheduled task and a .cmd for that, which
automatically stops SQL Server, 7z compresses and encrypts the database files
and source code repository, then move them to a usb flash drive, then restarts
SQL Server.

I never upload code to the internet, but I have copies of the repository in
multiple locations separated by at least 3000 miles.

Nightly I backup internet databases to a cloud storage provider. Nondatabase
files are stored simultaneously on local storage and cloud storage. The local
storage is mirrored to local snapshots hourly and nightly in case a roll back
is needed.

------
cullenking
I don't bother backing up the full system, just the user generated data. I am
storing lots of data as flatfiles rather than shoving them into the database,
so rsync helps there.

I like having a complete copy of all user-data on my home machine for
development purposes as well as backup. So, I rsync all user generated content
(they are 99% flatfiles so this is easy), as well as a daily mysql dump. My
other founder and I both perform this daily, so at any point we are only
exposed with a 1 day risk (seems acceptable at this point). This also gives us
two separate off-site backups, which is nice.

------
there
an openbsd vm in a... secret location... runs rsnapshot (rsync over ssh) to
all of my servers every few hours and keeps separate hourly, daily, weekly,
monthly, and yearly "full" backups. they are stored on a usb hard drive
encrypted with openbsd's softraid.

when the server boots up, i have to enter in the softraid passphrase, it
mounts the drive, then i launch screen and start up the script that spawns the
rsnapshot processes at the appropriate times. when starting it, i enter in the
common ssh key passphrase (which is restricted on all the servers to only
allow logging in from that network) and common mysql backup user password.
these are stored in memory and then the script runs rsnapshot every few hours.

once a day, the script does a full mysql dump of all databases on all servers
using a ruby script with net::ssh to tunnel mysql access over ssh. it does
this to keep "proper" mysql backups in case the rsync'd binary mysql files are
transferred in an odd state, and keep daily backups of everything. it also
lets me quickly get at data with standard text tools instead of having to
restore the raw mysql files somewhere temporarily, start up mysqld, and
extract it.

so once the server is booted, it's completely automated. i've been doing
backups this way for a couple years and have recovered files and databases
from them a number of times. no complaints.

------
njl
I have scripts (I use fabric, keep meaning to explore chef) that bring a fresh
OS install up to a production server automatically, doing all installation and
configuration. This is all in a DVCS system, so I have a local copy of my
repository as well as a hosted copy of my repository on bitbucket/github. That
takes care of code and configuration.

I have a cron job that dumps the databases into a directory with a time-
stamped name. The last few regular nightly back-ups are kept, then weekly for
a couple of weeks, then monthly for about a year. The parent directory is then
backed up to S3 using duplicity. I've done other, cleverer things in the past
with rotating through buckets based on the month, keeping every backup, doing
them twice a day, that sort of thing.

I test recovery by spinning up another server, running the scripts to bring it
up to production status, and running the script that uses duplicity to pull
the data dump directory back out of S3, and then restores the database from
the dump.

At $0.15 a gig to store data, and about $0.10 a gig to transfer it in, the
cost of S3 is pretty much just noise. I've got on-disk dumps of my database if
the database is corrupted on my machine, and I've got back-ups of those dumps
if I need them.

Duplicity is pretty fantastic -- definitely check it out.

~~~
rphlx
duplicity is awesome. can backup to gmail IMAPS, $0.00 per gig for storage and
$0.00 per gig for transfer in a high end GOOG datacenter. ;)

------
jbyers
No whole-system backups. Offsite (across the country) replicated MySQL
database with daily full snapshots and periodic encrypted dumps to S3. Real-
time encrypted mogilefs backups to S3 on a per-file basis. S3 MFA + versioning
is a huge win in both cases. rsync for a small number of exceptions: IMAP
data, development trees, etc. When a server fails, just re-image it, bootstrap
with puppet, start it back up.

------
mark_l_watson
I do a lot of simple things. Servers: use cron, zip, and s3cmd tools to backup
to S3; on EC2, use snapshots. Development laptops: use git repos hosted on a
rented VPS and I have a cron script to ZIP these repos and back them up to S3.

------
petervandijck
The data lives separate from the code: the code is backed up in Subversion,
the data lives in AmazonS3 (no other backup) for images, and a database in an
Amazon snapshot with daily backups automated.

------
oomkiller
Rsync.net, simple, cheap reliable, and helpful. If you call their telephone #
you get a real person, that knows WTF is going on, not some L1 support tech.

------
gibsonf1
EC2 AMI image -> S3 at milestones, mysql dbase backup to S3 every few minutes,
all files stored directly on S3, so only cached on server.

------
forkqueue
duplicity is great for off-site backups - full encryption, supports a wide
variety of destinations (amazon s3, FTP etc)

------
sogrady
We use ZRM to automate the backup and versioning of our MySQL databases.

Then we use cron and s3sync to mirror the relevant portions of the filesystem
to S3.

For those interested, the process is documented here:

<http://monk.ly/bFBd7w>

------
brandon272
Daily rsync to BQBackup, with once weekly offline backup.

------
buro9
Cron + rsync.net and their geo-replicated package.

------
BlueZeniX
ZFS auto snapshots, backup-server running incremental ssh "zfs send" |zfs
recv.

------
DanBlake
r1soft

