
 Explosion At ThePlanet Datacenter Drops 9,000 Servers, 7,500 Customers - nickb
http://www.allheadlinenews.com/articles/7011131199
======
metachor
This is an interesting lesson in disaster recovery planning for businesses
using third-party datacenters for hosting. One common scenario is for a
business with its own datacenter to use a private third-party datacenter (like
the one in the article) as their off-site DR location. But what do you do when
your DR location suffers an outage? Many companies will host in multiple
datacenters to prevent outages in any one physical location from bringing down
all of their operations. I suppose ultimate security could only be achieved
using a distributed approach, but that may cost beyond what many businesses
can afford (unless you are Google or Microsoft, for example).

~~~
SwellJoe
Theoretically Google or Amazon could provide the same level of reliability to
their hosted customers. But, in reality, uptime of servers at Amazon has been
significantly lower than The Planet, even accounting for this outage.

We have a couple of servers with The Planet, but luckily, they are in Dallas
rather than Houston. This event did make me rethink our backup procedures,
however. Backing up to another server in the same data center provides no
protection against this kind of event, and though it sounds like there is no
data loss, I'm sure a lot of folks were experiencing some pangs of regret
yesterday before the full scope of the damage was understood.

~~~
metachor
I think a minimal necessity for a DR plan is to have at least three
geographically seperated hosting locations with some sort of automatic
replication or fail-over. Again, this costs a lot of money. I don't know
whether the ec2/cloud-computing is comparable as of today.

------
bprater
Bring on the cloud. Enough of this hosting stuff on a "single" machine
craziness.

Folks in 20 years are going to scratch their heads when we try to explain what
we did back in the old days.

~~~
neilc
_Bring on the cloud._

A massive service outage is a reason to _encourage_ the move to cloud
computing? Such an architecture would only be more vulnerable to service
failures outside the user's control, not less.

~~~
johnrob
Not if your virtual instances can be replicated to another data center (an EC2
instance is location agnostic).

------
herdrick
More comments here: <http://news.ycombinator.com/item?id=205709>

------
herdrick
"They estimate they will be up and running by Sunday afternoon."

Still down.

------
dmix
Thats interesting, just yesterday I was reading into datacenters and how they
are engineered. The fire prevention techniques are usually really advanced and
that may be why none of the servers were damaged.

~~~
metachor
Indeed. Many data-centers have a multi-tiered approach to fire-suppression.
This usually includes a chemical suppressant as first line (often a de-
oxygenate that suppresses fires by removing oxygen from the room and does not
damage servers) as well as a preaction dry-pipe system as a backup (in this
case the fire has gone too far to save the severs and the goal is to protect
the rest of the building).

------
tlrobinson
The headline makes it sound worse than it was... no servers were damaged in
the explosion, power was just knocked out.

~~~
reggplant
Well it was powerful enough to knock down 3 walls surrounding the power room!

