

Ask HN: How do you backup your servers? - bmaeser

I do a lot of devops-work for different companies and "agencies", and i have seen a wide range of solutions so far. Unfortunately most of the existing backup solutions were really bad an i realized how less they even cared about it.<p>So HN: how, what, how often, where and when do YOU backup your servers?<p>examples:<p>configfiles, everything at /etc/ is under versioncontrol with git, everytime i make a change i commit and push it to a private server (offside) where every server has its own repo.<p>database, postgresql, every night at 02:00 a full backup with pgdump, triggered by cron, homebrew backupscript in bash, dump gets scp'ed to another host in a different datacenter, old backups are deleted manually when the disc gets full soon.
======
derekp7
I used to use a homebrew script that implemented rsync/snapshots (that is,
rsync data to a directory on another server, then "cp -al" all files to create
hard links into a directory called "server-datestamp-weekly" (or daily,
monthly, etc).

After running into scaling issues (esp. when needing to find a specific file
that I didn't have a date or server name), I wrote (and open-sourced) Snebu.
This works effectively like rsync/snapshots, but it stores the file metadata
in an sqlite database (so there is very little setup complexity /
maintanence), and individual files are stored compressed in a vault directory,
named by the SHA1 checksum of the file. This gets cross server file-level
deduplication for free. This is somewhat similar to how git stores files.

Just recently posted a 1.0 release, although I need to improve the
documentation a bit and post a few more front-end scripts (and finish the in-
app help pages). But would love to get some feedback on it.

The site is www.snebu.com if you are interested (this points to the Github-
hosted page, the project page is github.com/derekp7/snebu).

For data retention, I keep daily backups for a couple weeks, weekly backups
for 6 weeks, monthlys for a year, and yearlys for as long as I have space.

~~~
bmaeser
great project! so you call snafu still from custom sh-scripts and trigger them
by cron?

how do you backup your databases?

~~~
derekp7
Actually, it is "snebu", "simple network backup utility", but it does come in
handy in case of a snafu...

For database backups, I'm only doing Oracle DBs right now. The procedure there
is: 1) Back up all non-database files 2) Put the database in backup mode 3)
Backup the .dbf files 4) Take database out of backup mode 5) Force a logfile
switch 6) Backup the archive logs (archived redo logs) I'll have to make sure
I include a sample Oracle backup script (along with other DBs if I get a
chance to) either in the documentation, or in a samples directory.

So one of the features of Snebu is that if you re-use a backup name / serial
number (datestamp), it will append those files to the existing backup. As for
scheduling the backups, on my backup server I have a cron job that cycles
through all the hosts to backup up, and kicks off the client-side backup
script via ssh. So: backup-server -> runs master backup script, ssh's to
target host target-host -> runs backup.sh, which calls snebu via an ssh
command to the backup server

By having the master cron script run from the backup server (or actually any
centralized host), you can keep the backups from overlapping each other.
Currently, due to the way sqlite handles locking, running backups in parallel
can be a bit problematic (although I think I've got most of the concurrency
issues resolved, still have to QA my code changes though).

~~~
bmaeser
oops! VERY unfortunate typo. sorry for that. i think sigmund freud would have
its own thoughts on it :-)

i really like your approach, i have very similar setups (backup-master &&
remote execution via ssh) and put a lot of thought into this topic the last
couple of weeks. i think there is way to less love and tools who properly
handle backups. everyone runs his own scripts and tools, or buys some extremly
expensive and proprietary "stuff".

the whole point behind this thread was to get insight on other peoples system,
look at the pros and cons and write a library that fits 90%+ of the use cases.

as you can see by the amount of people, participating in this threat (by now),
no one gives a shit. which brings us back to snafu.

thanks for the insights and "snebu" thou!

~~~
derekp7
That's ok, you've given me a good tagline -- "Snebu comes in handy when you
have a Snafu". As for comment participation, unfortunately unless you luck out
and get a few upvotes immediately when you post here, you won't make the first
couple pages and no one will see your submission. I had the same issue when I
posted a "Show HN" about Snebu, no one saw it. I think the only way to get a
submission to be visible is to have a few friends up-vote you right away
(about 3 - 5 is all it takes to briefly make it to the front page), then if
enough other people like the content it will stick.

------
chsonnu
Chef

