
Yes, Rackspace Is Down And So Are Many Of Your Favorite Sites - vaksel
http://www.techcrunch.com/2009/06/29/yes-rackspace-is-down-and-so-are-many-of-your-favorite-sites/
======
fallentimes
We (<http://ticketstumbler.com>) use Rackspace's Mosso and were only down for
10 minutes.

Rackspace's twitter account just said: @Rackspace: All power is restored to
the DFW data center - all devices affected are starting to come on-line.
Details to follow.

~~~
PonyGumbo
Same here. Couldn't have happened at a worse time, but at least it was only
for 10 minutes or so.

------
callmeed
We have several servers with Rackspace and several hundred of our own
customers were affected by this outage (for about 30 mins).

For what it's worth, I still think they are the best managed hosting company
around. The customer service is great.

This is the only major problem I can recall since this one:
[http://www.techcrunch.com/2007/11/12/quick-plug-the-
internet...](http://www.techcrunch.com/2007/11/12/quick-plug-the-internet-
back-in-major-rackspace-outage/)

------
jme
We were down for over 2 HOURS! Not acceptable! Found out that one of our
servers was stuck at a BIOS screen waiting for someone to hit "Enter". Not
happy about it at all.

~~~
catch23
Are you using them as a colo? That's usually one of the first things I do when
I setup a machine to be shipped off to a colo -- make sure that a remote boot
doesn't require console interaction. Reduce the grub menu timeout to 5
seconds, always skip bios prompt regardless of errors, etc. Even if the colo
provides the ability to remotely interact with the console, it's less hassle
if you don't have to.

~~~
jme
No, they set up the servers for us. Didn't think to ask, "please make sure the
servers reboot without someone standing by to push 'enter' if needed". They
told me that the server's BIOS was saying something about the RAID battery
being charged and required someone to push 'enter' to continue. Thanks for the
suggestion.

------
shizcakes
I am a Rackspace customer (fortunately, in VA). Last week, they accidentally
sent emails out to many customers letting them know that there was heavy
construction (that they don't control) adjacent to their lot of land in Texas.
I'm betting that has something to do with it.

~~~
piramida
So what has really happened, or Rackspace does not go as far as explaining
customers what was the reason and what measures are taken so it won't happen
again? I'd be surprised if that is the case.

My current hoster costs less and does much more (you're informed and kept
updated in case anything remotely serious happens, especially if it affects
the whole DC - though happened only once during 6 years, customers had live
progress on the failure resolution).

------
quizbiz
I know nothing about hosting but how much more expensive can running fully
redundant be? Backups available at a separate physical location ready to go
online incase A goes offline? I guess the question is how much can businesses
afford to pay.

Has anyone done the math to see income lost per hour offline? (I'm pretty sure
Twitter makes money for being offline. heh)

~~~
vaksel
well if its fully redundant, it'd be 2x as much.

Personally I think, the best bet is to just have a backup on Amazon. Sure its
not as fast as switching to another dedicated server, but its just something
to have in case of emergency, users can afford to have the page take an extra
ms to load.

~~~
quizbiz
My thinking was, if they already own a datacenter in another location, why not
add another floor? Obviously it's not cheap but is it really 2x? If you're big
enough, you don't have to hire double the staff or buy double the resources
since you already have them.

~~~
jonknee
The expensive part about data centers isn't the square footage, it's the
HVAC/power/bandwidth/servers. So you have the space, that means you still need
to reserve twice the bandwidth, power, cooling and hardware. x2 cost. Actually
somewhat more because you need to account for all the bandwidth and time to
keep everything synced up.

~~~
lsc
at what rackspace can charge, new hardware is nothing. the problem is that if
I rent you a server, it's going to be pretty difficult for me to replicate
that server to another location without a lot of cooperation from you. the
more I screw with your server, well, the more likely I am to break it, too.

------
blhack
I saw my sliechost box go down for about 10 minutes. I'm really surprised that
they haven't sent me an email yet, they're usually RIGHT on top of that...

Anybody have any word on what happened?

~~~
mkull
You posted JUST too soon... 10 minutes ago I received:

\------------------------------------------------------------------------
Hello,

We have experienced an interruption in power to a portion of our Dallas-Fort
Worth data center facility. Power has been restored and our DC engineers are
working on devices that need to be manually brought back online as quickly as
possible. Further updates will be made as soon as new information is made
available. Please monitor our MyRackspace customer portal as this is the
quickest way we can get updates out to you.

Sincerely,

Rackspace

~~~
wglb
That actually sounds pretty scary. What sort of incoming fault trips (what
sounds like) DC breakers?

~~~
eli
If I recall, they were doing maintenance to their chillers this week. Hmmm....

~~~
wglb
Ah, so perhaps a little <http://en.wikipedia.org/wiki/Iatrogenesis>

------
zandorg
I had a laptop with 2 good batteries and I lasted out a power cut watching a
whole DVD. The ADSL 'modem' was powered by the phone socket and shared over
Wifi from another laptop with a (single) good battery, so I could continue
using the internet while watching this DVD!

I guess what I'm trying to say is that laptops might be a good solution for a
hosting company, because of the lower power requirements and cheap batteries.

~~~
Locke1689
That's not even remotely price feasible. Also, lower power requirements does
not necessarily mean efficient. In terms of power/watt, they have inefficient
power conversion and are particularly difficult to cool. Trust me, rackspace
knows what they're doing.

~~~
shizcakes
Actually, they aren't doing anything particularly amazing - my servers are
just Dell 2950IIIs.

Still, for one guy managing things, it's way easier than co-lo.

------
kragen
10 minutes is 1% of 17 hours, 0.1% of a week, 0.01% of 69 days, or 0.001% of
23 months. So they could maintain "five nines" if they had an outage like this
every two years — except for people like jme.

But of course outages follow a power law. On January 15, 1990, AT&T's entire
long-distance network crashed for 9 hours.

~~~
jacquesm
there was a _major_ outage less than two years ago iirc in the same data
center, a transformer blew up and shifted a wall. they're not going to get
their 5 nines for a long long time.

~~~
chops
Wow. That sounds almost as bad as the CIHost outage where thieves sawed
through the walls, stole a bunch of servers, and tazed the employee that
responded to the alarm.

I'm so glad I got my servers out of there only 2 months prior (after being
there for almost three years).

------
Dave_Kean
Cheers! May this source of so much spam be offline for 150 years.

~~~
_pius
What are you talking about?

~~~
sneakums
I guess he gets a lot of spam from Rackspace customers.

