

Rackspace: How to Royally Poop on Your Customers (8 Ways) - citricsquid
http://www.ericknopf.com/post/98678595715/rackspace-how-to-royally-poop-on-your-customers-8

======
leesalminen
3 hours is crazy! According to New Relic, my apps were offline for 11 minutes.

I agree with your points though, 9:20 PM on a Friday was a bad time to notify
customers. I had to write an email to all my customers at 11PM on a Friday.
Not only that, I had to call my business partner to read it before sending, as
it was a Friday night and I was inebriated.

Interesting story, after seeing the notice from AWS here on HN, I called
Rackspace on Wednesday night. It was late, and the gentleman answering the
phone said he worked in the NOC. He said that Rackspace was not affected by
the "Shell Shock" CVE, and that they would not need to perform disruptive
maintenance. Imagine my surprise Friday night.

------
jms703
Bad notifications, sure. But all you've really demonstrated is that you can
only handle planned outages. Planning around the unplanned outages are where
we earn our stripes (and paychecks).

You have servers running on infrastructure inside a datacenter that you don't
control. Surviving reboots and outages needs to be a part of your strategy if
these systems are important.

Oh, and there's this: [https://github.com/Netflix/SimianArmy/wiki/Chaos-
Monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey)

~~~
AznHisoka
Yup. However, that's hard when you've spent 99% of your budget on overpriced
Rackspace servers. Great brand ("nobody got fired for choosing IBM"), but
horrible bang for your buck. But hey, if it's not your money, can't blame them
for choosing them.

~~~
jms703
Sorry, but if you can do that (spend 99% of budget on servers), then you have
bigger problems. :(

------
underyx
>#1) Send a maintenance notice when no one will read it.

>#2) [...] Had we gotten notice on a monday, we could have switched servers or
companies altogether. Hard to do that on a weekend with short notice.

These two appear to be the same.

>#3) Be sure to NOT use the word downtime. Disguise the issue as a likely
small inconvenience.

>#4) Give Zero Expectations of Impact.

These two also appear to be the same. (These sentences themselves are actually
contradictory, but the point of the description in these two are the same.)

>#5) Put details in riddle format. The official email said, “Maintenance
window (all times CDT, UTC-5).” Whew! That cleared it up.

Why is specifying the time zone of the times given below a riddle? I can't
think of any way to specify exact times in a simpler manner. I'm really not
sure if 'When it's the middle of the day in some part of the USA' woul have
served customers better. I assume there were times specified, as the quoted
sentence wouldn't really serve any purpose otherwise.

>The Rackspace maintenance window was 6am - 6pm. Hmmm, thats odd.. thats
exactly peak time. Sounds perfect.

You have no way to know how much traffic Rackspace serves in that timespan,
Peak time for them might be way different than your peak time. They serve all
kinds of countries all over the world, you know.

>#7) Spin lack of information as “protection.” Certain details? Like estimated
impact? Progress estimates?

Most likely, they are trying to deceive anyone there, they should obviously be
referring to lack of information about the issue's nature, not lack of
information about maintenance work.

>#8) To pacify frustrated customers, just lie. I called into support 50
minutes into the outage and was told the migration was 85% complete. That
pacified me enough to get off the phone. Then two more hours passed.

I don't think Rackspace was actively trying to deceive people. This sounds
more like an issue with unexpected issues coming up, or the customer service
rep being poorly informed.

>#9 __Bonus Tip __) Direct customers to remedies that dont work. My support
gal was empathetic but at best, offered me the option to create a ticket so I
would be heard by those who care. Lets forget how impersonal that is for a
moment.

'Heard by those who care'? Again, I find it hard to believe that someone whose
profession is to resolve customer issues via phone would care any less than
people on the other side of the ticketing system. It is more probable that the
representative was just unable to assist any further.

------
Paul_Dessert
Not surprising. I use Rackspace at work. I've been called a "stupid asshole"
by a support technician (he thought he put me on hold to transfer me. He was
wrong). I've had security issues due to my "fully managed server" not being
"fully managed" and an array of other small inconveniences.

I've put up with it because it's a PITA to transfer servers. I think it's time
to start shopping around.

------
RowanH
What this shows is it's important to have a test server at a bunch of
different companies to see what's important to you and leave them there for a
3-4 months evaluating how they respond to issues, actual uptime, speed of
updating security mirrors etc. It's been a bit of an eye opener for me beyond
features/prices to see how companies actually perform where the rubber meets
the road.

~~~
underyx
Why would everyone need to have their own servers though at this age? Couldn't
we just have a common source for this information? Mind sharing your findings
here, at least?

------
jedicoffee
I understand that this sucks and their approach is terrible; However, if you
don't have redundancy you should expect downtime. Things fail. Maintenance is
a fact of life. Single points of failure are bad.

------
vblord
lol. I've heard some bad things about rackspace's uptime from someone else
recently. This article is actually very humorous in the way that it was
written, but is also full of very good points.

~~~
leesalminen
The only downtime I've had on Rackspace Cloud in the past 22 months was this
weekends maintenance (11 minutes total). YMMV.

