

Ask HN: Startup Disaster Recovery - bdonohue

Our company is in the process of raising a round, and one of our investors has asked us to put in place some disaster recovery procedures before investing. What DR procedures do you have in place for your company?<p>Some background:
* I am sole coder
* All code is on GitHub
* App is deployed to Heroku
* Domain names are on NetworkSolutions
======
eduardordm
Very 'interesting' they asking you this. Specially when it looks you are just
starting.

I run a credit card company on AWS, our website runs on a single, small aws
instance, but the POS/Financial servers are setup this way:

\- 8 multi-az oracle rds instances, 4 of them are slaves. It means we have 16
servers where 8 are manually standing by and 8 are on rds multi-az system. The
4 slaves are set to be used to recover from a data tampering event, not aws
outages.

\- Some dozens of app servers on the same region of each database. We use a
custom tcp/ip server I built in java almost a decade ago. We are thinking
about migrating to apache Mina.

My personal tips:

\- Why are you paying the heroku premium? Build you own aws servers, put nginx
and set cheap instances behind it. Let capistrano be your heroku.

\- Use some dynamic dns service, I would recommend dynect. Avoid amazon Route
53.

\- Use multi-az, set up the same database in another region (continent)

\- Get out of us-east

Build the very same setting into another region, create scripts to load the
slave country's database if possible.

If your database is too fancy or large:

Shard or distribute.

(Edit) Almost forgot: ALWAYS have fresh as possible, offline backups of
everything. Oracle is god when it comes to making this easy.

~~~
bdonohue
Maybe I've worded this incorrectly, but they're basically saying, "What
happens if you get hit by a bus?" My co-founder has access to one of our three
projects, but no one else has access to Heroku, GitHub, etc.

Also, the Heroku vs EC2 is another can of worms. Heroku is the better decision
for where we are now.

~~~
eduardordm
Incorporation agreements must have outlines on how to deal with such events.
Other than that, you need someone to replace you in case something happens to
you. That person should have the same access and info than you.

------
whichdan
Few thoughts:

\- Don't worry about code backups, assuming you have the following: 1\. Local
copy 2\. Local backups 3\. GitHub 4\. GitHub's own backups 5\. Copy on Heroku
6\. Heroku's backups (?) 7\. Any server backups

That's a hell of a lot of backups.

\- Make sure you can time how fast you can provision a new server and restore
backups. When you actually have to, it'll give you some peace of mind knowing
that your backups can be restored, and that you can reliably guess how long
it'll take.

\- Make sure your domains are "Locked" in NetworkSolutions, which prevents
transfer-out requests. I'm still mixed on whois privacy, but you're probably
okay using it with NetworkSolutions.

\- I've also struggled with figuring out what to do if I can't run the
business suddenly. For your domain, you can register it for several years or
set it up on auto-renew, so that's not really an issue. As far as
GitHub/Heroku, ... that's a tough question.

