

Rely your web startup on Rackspace Cloud? Think again - edwincheese
http://bencheng.net/rely-your-web-startup-on-rackspace-cloud-thin

======
matrix
I'm thinking Rackspace might well have been in the right on this one. If the
customer was in fact phishing, Rackspace was well within their rights to shut
down the account. It's really up to the application creator to prevent that
abuse.

That said, it's good to have a reminder of the risks of outsourcing your
hosting. I still think the tradeoff is worth it for experimental products
where you don't want to invest too much upfront.

~~~
nl
I agree. Hosting providers usually reserve the right to shutdown your server
if it has been hacked, and phishing is often more directly harmful to people.

~~~
watchandwait
Agree. This wasn't a simple DMCA issue-- phishing is an active, criminal
activity. Even one hour notice is generous.

~~~
A1kmm
Taking the contents of the article on faith, I think the point was that their
customers were abusing the startup's service for phishing.

This would be analogous to, for example, AWS taking reddit offline because a
user posted a phising link.

Nearly every web business which lets people put information on the web that
others can see will face abuse issues at some point or the other - and cutting
off a business and its legitimate customers because of one client misusing the
service does not inspire confidence.

------
malbiniak
This may sound contrived and sarcastic, but I assure you it is not.

Try your best to get Robert Scoble's attention on this. Rackspace has tasked
him with being an evangelist primarily focused on startups and bringing them
into the Rackspace services. It's my understanding that things like this
should be his primary concern.

I could be mistaken, but I'm used to that.

<http://twitter.com/malbiniak/status/22672201225>

~~~
jon_dahl
Your middle paragraph is reasonable and a good idea. What's with the first and
third?

~~~
malbiniak
first - didn't want to come off as "complain and your problem will be fixed."
i never anticipated having it called reasonable.

third - second paragraph is my interpretation of a conversation i had with
some people from rackspace and it could be wrong.

also, hello from minneapolis.

------
kjksf
What is missing from this article (and comments so far) is a more
comprehensive analysis of available options.

If I lease a server from linode or AWS or theplanet or serverbeach or ${your
favorite hosting provider}, would the situation be any different? I understand
the article's author frustration with Rackspace, but it's a single data point
hence hardly enough to be a basis for an intelligent choice of hosting
provider.

I'm not even sure if I sympathise with him. You can argue whether 1 hour
notice before disabling a server is enough or not but there is an obvious
conflict of interest.

The interest of the person hosting server, who can potentially be a phisher
himself, is for the site to stay up as long as possible.

The interest of the public is served by terminating the server as quickly as
possible.

~~~
lsc
Your last two lines sum up the problem nicely.

Everywhere, the more you pay, the more effort they will put in to helping you
clean up your messes rather than shutting you down right away, which makes
sense, because helping you clean up your mess is an expensive business to be
in.

~~~
chopsueyar
Which doesn't explain Rackpace's behavior, considering their premium cost for
"fanatical" support.

Perhaps Rackspace cares more about it's shareholders than customers?

~~~
lsc
I could be mistaken, but I think that the rackspace cloud is their cut-rate
product which comes with less support.

~~~
chopsueyar
[http://www.rackspacecloud.com/cloud_hosting_products/servers...](http://www.rackspacecloud.com/cloud_hosting_products/servers/support)

"We talk AND we listen. We're big fans of transparency and feedback.
Technology works so much better combined with a bit of humanity."

------
ohashi
I am really not a fan of all the defending RackSpace here. They pulled the
plug unreasonably without proper notification.

If you're going to pull the plug or even thinking about it... email simply
isn't going to cut it. You need someone to call the owner and make contact to
explain what's going on or how to resolve it. I've had my servers compromised,
I've had phishing content setup before, I have never had the plug pulled. I've
had hosts contact me, give me appropriate amounts of time to handle it and
some of them even offered to help secure my box or look into how it got
compromised in the first place.

------
varikin
All this talk about how Rackspace should be treating startups make me think,
"Why?"

I understand startups may not have the personel to react as fast as a larger
company with dedicated personel, and I understand that there might be a very
large percentage of startups on Rackspace which might account for a nice chunk
of revenue.

But what makes startups different from my personal website or a larger
corporation? If Rackspace receives a complaint about a phishing site hosted on
their servers, they should do what they can to correct that, regardless of
which client is using the server that has the phishing attack. The startup
should get the same treatment from Rackspace as the large corporation and the
guy with some simple homepage.

~~~
edwincheese
It is like asking why we need special facilities in building for the disables.
Why didn't we treat everybody the same and don't provide any special
facilities for disable?

I do think that site owner of any scale IS responsible to abuse complains, but
the scale of the company do make a different. The influence and harmfulness of
a fake shop in Amazon is not same as a fake shop in a very-small-online-shop.
Thus people expect Amazon to take action instantly.

A start-up is not possibile to react as fast as large company, so why
shouldn't we give them a reasonable time for them to do their job?

~~~
wrs
The people being phished don't care how big you are or how fast you can
respond. What would _they_ consider a reasonable time?

~~~
chopsueyar
If the people being phished know they are being phished, then why are they
allowing this?

------
cedsav
Welcome to the joys and sorrows of hosting user-generated content. If this
makes you feel better,rackspace is just your first problem. You'll also find
soon enough that abuses can get your domain blacklisted in chrome, firefox,
and for opendns users.

------
mgkimsal
What _should_ happen here is this...

When a server is 'shut down' for phishing or spam, a firewall blocks all
incoming/outgoing except for traffic to/from a pre-determined IP address along
with the notice that the server is being quarantined.

Site owner/admin can then access the server, perform any investigations or
deletions necessary, notify data center, then data center opens traffic again.

Alternatively, something like a web-based shell allowing access, but all other
traffic denied, would be acceptable.

I've had servers shut down for 'abuse' which was one complaint from someone at
1am local time for me. I supposedly got a call from xxxxxx at 2 am, notifying
me that action would be taken, and my server was taken offline at 3am. I
wasn't awake until 7am, and couldn't get things resolved until about 10am. I
was told I needed to 'rectify the situation', but how can I do that when
access to the server is blocked? It's a ridiculous execution of policy, and
only serves to heighten everyone's frustration. A private web-shell or single
IP in the firewall to allow access would resolve most of the ill-feelings site
owners caught in this situation have had.

~~~
lsc
My policy is that I shut down the server, I create a fresh server for the user
and then I attach the old disk read-only to that server, in case the user
needs to retrieve data.

The problem is that there is no way to 'clean up' after a break in without
booting from trusted media. Otherwise, there is no way to know if you have
closed all the backdoors the attacker left.

------
Aegean
We had a similar problem with another vps provider. Some php script was
remotely exploited and sending spam. They disabled the VPS. Our website and
email was down for 24 hours. They enabled the VPS the next day for us to look
at it. What's worse I was asked to reinstall the VPS and it took me some
convincing for them to enable it so I get my data from the server.

Lesson learned? I think you should never have any operations where you don't
have full control. This still holds true but it was a bit ironic because to
have full control I ran everything on the VPS, effectively transferring full
control to the VPS provider! It was the single point of failure.

Now I will transfer each different service to a different provider. Email to
rackspace, websites to a host, git services to github and so on.

------
goosmurf
Post is lacking information on whether the site was actually a phishing site
and who (or which entity) submitted a complaint.

I think verification of the legitimacy of a complaint should be a critical
step before disabling a site, otherwise you're prone to DoS.

It would also be good to know what steps the complainant took. Did s/he try to
contact Pandaform, or immediately go to Rackspace as the owner of the IP?

Without knowing whether the complaint was legitimate, and what steps Rackspace
took to verify this (or not) its tough to say whether their actions were
appropriate.

~~~
chopsueyar
Their actions are inappropriate.

The majority of commentors here assume guilty until proven innocent.

In fact, the author of the article has still been unable to identify the
"phishing" form. Was there even really a phishing form?

With all this anti-phishing technology built into modern browsers, why is it
Rackspace's responsibility to "protect" users from "alleged" phishing sites?

For copyright, we have the DMCA (for better or worse). Perhaps we need some
similar sort of due process for hosting providers or cloud providers.

Most appalling is Rackspace's lack of transparency in handling this.

Pandaform was never contacted by the complainant. The complainant only
contacted Rackspace. Rackspace assumed, without investigation, the complaint
was legitimate and gave the guy notice via email.

Also, the real rub, is the fact the Rackspace would terminate his account if
he got a second complaint.

So, it is not necessary to have an actual phishing form on his site to have
his Rackspace hosting terminated. Someone simply needs to allege this to
Rackspace, and his account and data will be gone forever.

That is fanatical? Again, I cannot recommend Rackspace.

------
lsc
this will happen at any responsible web host. If you are hosting phishing
sites, expect to get taken down. This trickles up. if you run a hosting
company, and you get enough complaints that you don't deal with, then yeah,
you will get shut down or asked to leave.

That said, I think especially for higher-priced services, a phone call would
be nice. (Note: I don't call my customers, though this is a policy I've
considered implementing.) I'd be interested in what other people think about
other notification systems.

~~~
oomkiller
I don't think the issue is that he was hosting an active phishing site. The
main issue here is the amount of time he was given to fix the problem was too
small. You think Rackspace's upstreams would shut the pipes down if there were
a bunch of phishing sites that set up shop? Doubtful. Usually they only get
involved when there is a MASSIVE DDoS.

~~~
lsc
Well, from my experience in industry, big providers of bandwidth do apply
pressure (up to and including the threat of disconnect) to large hosting
companies in an effort to get them to clean up their act, even when it comes
to things like spamming and phishing that are less outright destructive to the
network than DDoS activity. You are right that a million dollar customer will
get a lot more slack than a twenty dollar user... but, uh, to me that should
be expected. Dealing with abuse is very expensive. Some places I've worked
that has been the majority of support costs.

------
fido
Rackspace (and others) need to call their clients on the phone in these
situations. They feel it is so important that it must be taken care of in one
hour, yet they use email. Not exactly fair to the client....

~~~
lsc
Isn't the rackspace cloud their cut-rate service? would they have called if it
was a slicehost (premium price) customer?

Still, I think it is a good idea, and one I ought to implement in my own
service. The problem is that we all hate dealing with the phone, but that
sounds to me like a lazy answer.

On the other hand, really, if you are an abuse problem, honestly, I don't want
you as a customer.

~~~
epochwolf
Slicehost doesn't have phone support last I checked.

------
credo
imo the biggest mistake here was in PandaForm trusting all of its user-
generated content.

It seems like they should have been doing some sort of verification of the
UGC. Alternately, if they really wanted to go with a fully laissez-faire no-
security-checks approach, they shouldn't have picked a hosting company (in
this case, Rackspace) that requires customers to sign an agreement that has
strict anti-phishing rules.

------
chopsueyar
I second this. Not a fan of Rackspace.

Some may argue having your own hardware is more expensive to maintain, but
there is a definite advantage to controlling your physical hardware.

Rackspace is helpful until you have a real problem, and you are left to fend
for yourself.

------
fookyong
How was this construed as "phishing"?

Did the customer simply set up a form that asks for a user's email address?
because if so that describes tons of other services out there... e.g. Wufoo,
Google Documents (but I guess they are not hosted on Rackspace!)

~~~
rickmak
FYI, my phishing form @wufoo: <http://rickmak.wufoo.com/forms/phishing/>

Look professional? ;) Similar form on pandaform will make pandaform shutdown
entirely.

~~~
fookyong
it looks like it got taken down. what was it?

------
oomkiller
Good to know I wasn't wrong when I switched to Linode (From Rackspace). While
phishing and spam is damaging, owners need more than 1-2 hours sometimes,
especially when they are in a different part of the world.

------
PonyGumbo
Oy. This comes the day after I moved all of my sites to Rackspace Cloud.

~~~
pinksoda
Oh no, get as far away as you can!

Check out Linode or SoftLayer.

------
royuen
@matrix, I think overall direction is correct, Rackspace should try their best
to notify their customer if they received complaints. However the ridiculous
point of this case is totally shutting down all servers of a startup in just 1
hour, which is really horrible. Instead I believe they should at least give 12
- 24 hours for a startup to react and investigate before taking down all their
services?

Is that really on the same direction that they want to serve startups?

~~~
damncabbage
> they should at least give 12 - 24 hours for > a startup to react and
> investigate

By which time the phisher has already done the damage and moved on.

------
lmz
I wonder if this abuse monitoring service can be outsourced somehow. Pay a
team of people around the world to monitor the abuse email address and hand
them the kill switch for so they can respond immediately to take down one
account only instead of the hosting provider taking down the whole server.

------
paolomaffei
Linode isn't really any better. I sent an email campaign to an optin list from
campaignmonitor - someone (1 person out of 1000) reported spam.

Campaignmonitor asked us what was going on - while this is debatable they may
have been damaged from the complaing

But even linode asked us about it, just because the website where people
signed up was on a linode hosting.

1 complain, nothing to do with hosting a part that the link is the here, and
they asked us if there was something wrong.

I wonder if you can just blast a spam email with a competitor website (hosted
on linode) inside and get him offline...

------
mcknz
To be fair, a service the size of Rackspace Cloud is bound to screw up at
least a few times.

Then again, no one ever got fired for switching to Linode.

~~~
StavrosK
First of all, what is up with Linode's London datacenter? I barely press enter
on commands and they've already been carried out. Seriously, it installs
packages before the text from the server reaches me, and I get 15 ms latency
on it (the SSH echo feels faster than my local PC).

Is it just me, or are their other datacenters slower than their London one? It
might be the latency, as I was actually _in_ London, but the one I have in
Georgia doesn't seem to have that fast disk accesses... Bottom line, though,
the London datacenter is just time-traveling fast.

