Hacker News new | past | comments | ask | show | jobs | submit login

Although I sometimes get downvoted for this, I'll say it again:

You can't outsource your liability.

If your product is a webapp, then the underlying messy bits of backups, hardware, availability and redundancy also require some amount of conscious thought on your part. Not every site/app needs it's own mini-datacenter, and you might not even need your own dedicated server (though you probably do when you reach a certain minimal amount of scale). But you DO need to have someone who is thinking about backups and availability, and a valid solution is not to assume that the smart folks at Amazon or Rackspace or any other hosting provider are going to be completely and consistently working with your best interests and uptime in mind.

EVRYTHING fails at some point. Every server, every generator, every upstream connection, every hosting provider big or small. And in this case I mean fail as in goes dark for some period of time not covered by backups or hot-spares.

So, plan accordingly.




You can't outsource your liability.

Of course you can. That is the entire reason the insurance industry exists.

More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own only to discover that I really suck at thinking through all of the design challenges of backup solutions. (Something which always seems to get discovered that the most inconvenient of times.)


What is your "insurance" for hosting? I'm not aware of any SLAs that actually pay out anything near equal value to an outage they caused. If you're paying a hosting provider $500/mo for a sever that you generate an average of $1000/hr in web sales from and they have a 3 hour outage, you'll get an SLA credit for a couple of bucks applied to your next months service.

The Internet is filled with news about hosting provider provided backups being unusable for a number of reasons at inopportune times.

You should have a backup copy of your code/databases in your control, on a machine that is completely independent from whatever you are doing your production hosting on. You should have done a "warm metal" install and test of that code another server to be sure that you can recover operations in a reasonable amount of time (whatever is appropriate for your case).

For your scale (based on your posts here, my assumption: Single developer, or developer with a couple of contractors; production site; 1-5K visits per month; non time-sensitive/mission-critical service.) you probably don't need high-availability auto-failover. But, you SHOULD have your DNS hosted separate from your hosting provider, you SHOULD have low TTL's, you SHOULD have a backup server in a warm state at some other provider, and you SHOULD know how to at least do a basic DNS update to redirect traffic over to a backup site that either runs the service or puts up a basic, friendly "OOPS, BRB" page.

I've often thought of a startup that would basically human-automate these things for guys like you. You still wouldn't be 100% self-sufficient, but you would be able to outsource SOME of your entire reliance on 1 provider. You'd then have to have 2 tiers of total failure (your provider, and this service) to encounter complete down time.


> More practically for the instant case, I use a provider who has a turnkey backup option, rather than one which would force me to spend expensive engineer time rolling my own

EBS lets you create backup snapshots in S3 with one single command. The problem the (I think the epithet is warranted here) idiot who posted the original article had is that he didn't use it.


10 years later you realize you aren't covered for that cancer surgery because you didn't check off the box saying you were a smoker on the 100 page form.


I agree with this 100%. It seems that an awful lot of companies like to pass the blame as soon as something unwanted happens. Yes, it may be the underlying fault of a service provider but, to your customers, you are the only company they deal with and it's your reputation that will be impacted no matter how many posts you put on your site and all over the internet.

Using service companies does not excuse poor customer service.


This could well turn out to be a fatal flaw for enterprise adoption of third party clouds: when corporate IT staff screw up like this or just messing with a control panel (check the AWS forums where too often a "It disappeared!" complaint has a reply of "It was deleted at thus and so time and date using the control panel."), are they going to accept the blame or shift it to the third party?


This already happens every day when the blame is passed through to Oracle, Gartner, etc. without any acknowledgement of the fact that, bad as the software truly is, it was selected and purchased at great cost and [usually] considerable delay by the people disclaiming responsibility.

In other words, this will only be a fatal flaw if the business customers stop giving IT departments a free pass for being bad at technology.


That's true, I agree 100%.

I'd add to that though, if you're a small company sometimes you must recognize that a larger 3rd party can use scale to provide a more reliable service than you can.


"You can't outsource your liability" I have to disagree. You can define any part of your business as a "liability", so in that case you don't have to outsource anything at all. In fact a big reason people outsource tasks is because of liability. AWS promises physical redundancy, which apparently doesn't mean crap!


Saying it "doesn't mean crap" is an exaggeration. AWS's physical redundancy offers a marginal increase in reliability (roughly 3.5%, according to their stats) at potentially lower cost than providing the same system in-house because of economies of scale. They never promised to be 100% fail-safe (and would be foolish to do so.)


The point is that they are not backing up your data they way they advertise it. The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist. Also if adding physical redundancy improves reliability only by 3.5% it means that they have different definition of the term.


The whole point of physical redundancy is to eliminate single point of failure yet from their email it seems that such single point still exist.

Which bit of "multiple failures of the underlying hardware components" leads you to think that a single point of failure still exists?


AWS promises you physical redundancy, what they don't offer you is an ironclad guarantee that nobody, customer included will ever screw up. So you need backups. It's really simple.


Amen.

It's quite amazing how many people think they can point the finger at some third party because they 'handed it off'. After all, if it's amazon then it's safe as the bank of England right?

But even banks can burn down and even amazon does not owe you a penny if they lose your stuff.

They'll say sorry, really nicely and maybe they'll offer you some store credit, comes in great when you have lots of free time, you can get some books. Maybe on how to back-up your data or so.

But if you are halfway competent you smile and take that backup and you continue your business, because the value of your business outweighs amazons liability by a huge factor.

Backing up is risk mitigation, if the risk to you is small then you can afford to live without backups, it means that the cost of backing up outweighed the cost of re-creation of the data.

If the risk is larger then you probably should accept that, and go do something about it.

Either way, the responsibility is yours.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: