
Why I Won't Use Rimuhosting - joao
http://www.aaronsw.com/weblog/rimuhosting
======
SwellJoe
Several of our customers are hosted with Rimuhosting, and we've interacted
with them on a number of occasions because of that. In my experience they are
very responsive, knowledgeable, and helpful. That's not to say this isn't a
pretty irritating set of events...and silently shutting down Apache shouldn't
be anyone's first instinct.

But, I've had bad experiences with literally every single host I've ever used
(dozens of them), and the longer you stay with a provider, the more likely you
are to find a tech having a bad day, or their network fiber getting cut by the
utility company, or various other demonstrations of man's fallibility.

I think the good probably still outweighs the bad with Rimuhosting...as Aaron
mentioned, they do have a good reputation among many people who ought to know.
The horror stories I hear about _really_ bad hosts leave this event in the
dust in terms of sheer stupidity and lack of good intentions. If this is the
worst you've ever experienced with a hosting provider, count yourself very,
very lucky. The data loss stories I've heard, in particular, are enough to
keep me awake at night.

In the interest of full disclosure, I'll mention that Rimuhosting is a
customer of ours (along with thousands of others). So, if you reckon a
commercial relationship worth a few hundred bucks a year would sway my
opinions, feel free to take this comment with a grain of salt or not at all.

~~~
apotheon
Gaining root in such an underhanded manner against the client's wishes is
bordering on criminal behavior. It doesn't get much worse than that.

Of course, I write about security professionally. Maybe I have a different
perspective on what constitutes unacceptable behavior than you have.

~~~
SwellJoe
_Gaining root in such an underhanded manner against the client's wishes is
bordering on criminal behavior._

No. Not even close. Rimuhosting owns the box in question. You can't possibly
break into your own box.

 _Of course, I write about security professionally. Maybe I have a different
perspective on what constitutes unacceptable behavior than you have._

I work in the hosting industry professionally. I've never been or worked at a
service provider, but I've worked for the industry for 12 years as a vendor.

As a security professional, perhaps you can take the other side's view for a
moment...

What would you, as a security professional who was dumped into the role of
running a hosting data center, do when a box on your network (which has
thousands of other boxes) has been rooted and is sending out spam at a rate of
a million messages per day, even if you _know_ that your paying customer is
not responsible for sending those messages? This much traffic is impacting
network performance for others, is spewing filth into the mailboxes of
innocent people all over the world, and your customer is unable to stop it or
cannot be reached in a reasonable time frame. What do you believe your
responsibility would be in such a case?

How about if it is (innocently) DoSing other systems on the network? This
happens a lot more than you'd think.

Many people here are suggesting that simply applying resource limits would
solve this problem...but that exhibits a lack of understanding of how such
resource limits work, and the likely end result for the customer of such
resource limiting in the event something has gone wrong. To the customer, a
limit on resource usage could very well also result in services becoming
unavailable. If you cap CPU at 10%, and your website gets enough traffic to
need 90% of the CPU, you'll only be able to serve a small percentage of your
clients (and they will wait a long time). The same is true of nearly every
resource limit in a hosting environment. If you are using more than your fair
share (which I know Aaron has explained he was not; again, probably poor
judgment and problem assessment on the part of the tech) then when the limits
are imposed, you will be "turned off" for some of your users. You needed those
resources to serve all of your requests...you don't have those resources
anymore, so you can't serve all of those requests anymore, or if you do, you
do so very slowly.

As programmers, you all know there are no silver bullet for solving hard,
complex problems. Resource usage in a virtualized environment with untrusted
users is a hard, complex problem. No amount of hand-waving about your "rights"
makes that less true.

~~~
apotheon
> No. Not even close. Rimuhosting owns the box in question. You can't possibly
> break into your own box.

I don't think you're familiar with the common uses of "bordering on".

> What would you, as a security professional who was dumped into the role of
> running a hosting data center, do when a box on your network (which has
> thousands of other boxes) has been rooted and is sending out spam at a rate
> of a million messages per day, even if you know that your paying customer is
> not responsible for sending those messages?

I'd stop the server _without_ gaining root against the client's wishes,
immediately inform the client, and try to get things cleaned up to everyone's
satisfaction (except the security cracker who rooted the box, of course). I
_wouldn't_ gain root on the box in violation of the client's obvious wishes,
particularly after articulating specific policy saying that you can remove my
initial root access to the box if you like, and shut down specific services
that are making my life difficult for reasons entirely unrelated to the
client's running of the site, then send an email later pointing out my policy
that fails to mention I might do this.

Privacy is security, after all:
<http://blogs.techrepublic.com.com/security/?p=293>

Nothing you say in your hypothetical examples in any way justifies _accessing
the customer's data_ , particularly by _breaking in_ to the system. Shut it
down? Sure, if circumstances call for that. Start meddling with server
configuration and establish what amounts to a rootkit on the system (in
effect, if not in technical truth) without the client's permission? That's
just shady.

> No amount of hand-waving about your "rights" makes that less true.

Perhaps you should stop waving your hands about that straw man, then.

~~~
SwellJoe
_I don't think you're familiar with the common uses of "bordering on"._

I'll promise to read up on it, if you'll learn the meaning of "not even
close".

 _Nothing you say in your hypothetical examples in any way justifies accessing
the customer's data, particularly by breaking in to the system. Shut it down?
Sure, if circumstances call for that. Start meddling with server configuration
and establish what amounts to a rootkit on the system (in effect, if not in
technical truth) without the client's permission? That's just shady._

I will merely mention that most hosting provider customers not only accept
this kind of thing, they _demand_ it. If you spend a little time on web
hosting forums (as I must because it is my industry) you will notice a very
strong tendency for complaints to be system administration related. The
customer expected _more_ involvement than the hosting provider offered, and
thus things went horribly awry.

I agree with you, _entirely_ , that if you ask the host to never login to your
system, they should respect that wish. But, I can also state without
hesitation that you and I (and most people here at HN) are thoroughly in the
minority in wanting our hosting provider to never login to our hosting
systems. The default mode for hosting providers is to drop in on the box
within a couple of comments in their ticketing system...if it can't be solved
with one or two replies, then it's safest to simply drop in and fix it. For
most hosting customers this is not an invasion of privacy or "breaking in", it
is "great support".

Finally, as a security professional, I'm sure you're also aware that with
access to the hardware, your host _has_ root all the time. There is nothing
you or I can do about it. Even more interestingly, the host also has the
ability to login, poke around, and _never tell you about it_ (and not leave a
trail...just boot up a live CD, mount up the disk read only, and poke around
til their heart's content). Also nothing you or I can do about that. With
someone else having access to the hardware, you have nothing but good faith on
the part of the hosting provider.

~~~
apotheon
> I'll promise to read up on it, if you'll learn the meaning of "not even
> close".

I don't see how "I know you are, but what am I?" is a very strong argument.

> I will merely mention that most hosting provider customers not only accept
> this kind of thing, they _demand_ it.

Give it to customers who want it. Don't ask someone if he wants it, get a "no"
answer, then turn around and do it anyway. The former is good customer
service. The latter is shady and underhanded.

> The default mode for hosting providers is to drop in on the box within a
> couple of comments in their ticketing system...if it can't be solved with
> one or two replies, then it's safest to simply drop in and fix it. For most
> hosting customers this is not an invasion of privacy or "breaking in", it is
> "great support".

Most hosting customers don't have explicit suggestions from the host that if
they don't want the hosting provider logging in to the system they can remove
their SSH keys -- and, more to the point, most hosting customers don't do that
then find out the hosting provider's support personnel have been rooting
around (pardon the pun) in their data anyway.

It's "breaking in" in this case _only_ because the SSH key for access was
removed, with the hosting provider's blessing, and they basically leveraged a
local access vulnerability to give themselves root access.

If the hosting provider had a policy that forbade customers from obstructing
host administrators from logging in to the machine, articulated in the terms
and conditions of use, I'd say go for it -- but that's not the situation in
this case _at all_.

> Finally, as a security professional, I'm sure you're also aware that with
> access to the hardware, your host has root all the time.

In principle, sure -- but when there's a clearly encouraged expectation of
privacy, it's _really_ bad form to break in to the system against the
customer's wishes by virtue of having access to the hardware. That's a
betrayal of trust.

> Even more interestingly, the host also has the ability to login, poke
> around, and never tell you about it (and not leave a trail...just boot up a
> live CD, mount up the disk read only, and poke around til their heart's
> content).

Indeed. The fact it wasn't kept secret in this case just shows how little they
value the customer's request that they don't log in to the system with root
privileges and muck about with the customer's data. I didn't say they were
necessarily _malicious_ about it -- but that doesn't mean it's not _bad_.

> With someone else having access to the hardware, you have nothing but good
> faith on the part of the hosting provider.

. . . or, in this case, bad faith.

------
Maro
Something similar happened to us a few days ago on our RimuHosting VPS. A
Python script was eating ~80% of the VPS CPU, so their tech guy, without
asking us, tries to login as root to our box (WTF?). He fails, so he responds
by rebooting our VPS, thus (?) causing some disk corruption, which he then
tries to repair. Then, after-the-fact, he writes us a very confused email
about the whole story.

Needless to say, we're moving away from RimuHosting. This is unacceptable.

An excerpt:

"I noticed your VPS was using a lot of CPU. I was not able to log in there,
and on the console I saw lots of out-of-memory messages.

I restarted the VPS to make sure it was running in a sane state. I checked the
console after a reboot and saw an error indicating some filesystem corruption.
I stopepd it again and repaired that. Then replaced the /etc/inittab file
which appeared to be corrupted. Now I see that has booted fine."

------
sho
This article reminds me of Hard Disks. Talk to any "PC fan" and they'll regale
you with a story of how they will never, ever use $HD_BRAND after it crashed
and lost all their data. Thing is, the brand changes depending on who you talk
to. And a real professional will tell you that all the brands have similar
failure rates, not to put your faith in any hardware, and advise you on a RAID
and backup strategy.

Same here. I know people who use rimuhosting, they seem happy enough. I use
Linode personally but have no real preference. This article smacks of a one-
off bad experience which could happen anywhere - and doesn't even seem that
bad. It's a VPS, for christ's sake, if you don't like it get a dedi or even
better a colo.

~~~
apotheon
The reason I don't use consumer-grade Western Digital drives has nothing to do
with a specific drive failure event, and everything to do with the fact that
consumer-grade drives from WD are basically factory seconds of WD's
enterprise-grade drives.

~~~
Confusion
Which is exactly the kind of story that gets perpetuated over and over again,
with different $brand's substituted. It doesn't even make sense: what
criterion is used to seperate 'enterprise' from 'consumer'? They can't
possibly test every drive (too expensive) and if they watch production
parameters and decide based on those, chances are that most consumer drives
are 'enterprise' as well. I find stories like these extremely implausible.

~~~
apotheon
A nontrivial percentage of the "consumer" grade drives (basically, the stuff
you can get at Best Buy) were actually manufactured as higher-capacity drives
for the premium line (I forget the model line term WD uses at the moment) but
had bad sectors coming off the line. They marked off those sectors, rounded
down, low-level formatted so it only reported the "new" size, and sold them as
lower capacity drives for retail consumer sales. Is that a more helpfully
precise explanation?

~~~
Confusion
That explanation still leaves some questions open, like: is there a reason to
suppose that a drive with 'bad sectors coming off the line' actually has a
higher expected failure rate? Or is it actually more like what happens with
CPU's, where a batch may 'fail', but will still yield CPU that can be sold as
'lower' CPU's than the batch was originally supposed to deliver.

Marking something 'enterprise' grade is often a placebo and has more to do
with service, contracts and well-marketed expectations than with actual
differences in the goods involved.

~~~
apotheon
> is there a reason to suppose that a drive with 'bad sectors coming off the
> line' actually has a higher expected failure rate?

If it doesn't hurt me to choose a different brand or a different model line,
the more relevant question becomes "Is there sufficient reason to believe that
a drive with 'bad sectors coming off the line' _doesn't_ have a higher
_actual_ failure rate?" It's better to be safe than sorry, as they say, and my
experience is that once a drive starts degrading, it keeps degrading.

The reason CPUs can often be sold at a lower capacity than originally intended
without worrying about an increased likelihood of later failure is because
transistors are discrete devices; if half of them fail, as long as the rest
can still be accessed, you just have a CPU with the same architecture and half
the transistors. On a hard drive, however, permanently hosed sectors represent
actual problems on _part of_ a single discrete device -- the hard drive
platter -- and it's entirely possible that the rest of the device may be
affected by this. Magnetic areas on a platter are quite so distinctly
segregated as transistors on a chip.

> Marking something 'enterprise' grade is often a placebo and has more to do
> with service, contracts and well-marketed expectations than with actual
> differences in the goods involved.

True -- but when you explicitly use drives that have failed the quality
control required for "enterprise" drives as your retail consumer grade drives,
that kinda changes the landscape a bit.

------
edgeztv
In my 2+ years hosting typeracer.com with Rimu, they have been absolutely
outstanding. All my questions, no matter how complex, were answered within
10-30 minutes. Hundreds of times. Anything I've asked them they did. They
installed all kinds of packages and performed configurations I requested, all
for free and without hesitation. And I'm not a premium customer by any
stretch. I only have 2 quarter-VPS instances with them for about $200 a month.

In the 2 years I only experienced 20 minutes of downtime recently due to a
one-time network hardware upgrade (1 week's advance notice was given - a bit
short but forgivable) and another 20 minutes due to a DDoS on their network
about 6 months ago (which most likely prompted this recent hardware upgrade
:).

I really hope people don't jump to conclusions from a single data point
(aaronsw). I would recommend RimuHosting to everyone I know without
hesitation.

------
olihb
I'm with Rimuhosting since forever.

Their support is unbeatable. Many times they helped me(I'm a dev. not a sys.
admin.) for free where other companies would have charged me.

I agree that adding their key on the VPS without talking to you sucks but
having a 100% utilisation of your CPU is a bit egoist. That means that nobody
on the host machine can burst their share for a quick high-cpu job.

If you want total control over your machine, get a dedicated box. Even then,
some hosters demand root access on your box.

~~~
tdavis
_nobody on the host machine can burst their share for a quick high-cpu job_

That only makes sense if there is only one other VM on the machine. Which is
basically never the case.

EDIT: Or they use a pedestrian virtualization layer.

~~~
olihb
I'm not sure I understand.

If there are 8 VPS on a machine and I'm always using 1/8 of CPU time, nobody
else can't have access to this 1/8 share. I know that 1/8 of the CPU share
"belongs" to me but it's a shared box and it's just being a good neighbour. I
appreciate it when I need to run a quick high-cpu on my VPS and I can use more
than my 1/8 share of CPU.

~~~
jonknee
You'd need seven "bad" neighbors for this to be the case.

~~~
olihb
You're right but in this case the user is a freeloader.

He can use more than his 1/8 share of CPU time but others can't use his 1/8
share.

~~~
tdavis
Who cares? He should only be using the other 7/8s when nobody else is; that's
the entire point. If their virtualization isn't setup to keep people from
bogarting resources on a "first come, first keep" basis, that is their poor
decision to live with; they shouldn't be taking it out on customers by killing
processes and so on.

A VPS host shouldn't be asking or telling customers to manage their own CPU
resources, period; that's the point of a VPS in the first place. It's a
virtual _private server_. It _should_ be isolated in a fashion such that even
if I'm using 100% CPU 24/7, it doesn't affect anybody else. If a host told me
to "use less CPU" I'd tell them to "acquire better virtualization" and find a
better host. Ridiculous.

~~~
olihb
I agree but you should not be using a VPS if you are a high CPU user. You use
a dedicated server for that.

Most shared providers will charge you more if you use more CPU or require that
you upgrade to a dedicated server. I'm pretty sure that any hosting company
will be happy to lose you as a customer if your usage degrades the other users
performance.

~~~
jonknee
Being able to use up to your share of a VPS is why you pay for a VPS and not
shared hosting. If you pay for 1/8th of a server you are entitled to use 100%
of that quota 24/7/365. Using your resources doesn't "degrade" other
customers' performance unless the server is set up incorrectly. As long as
they can access their 1/8th everyone should be happy.

------
neilc
_They ... offered to take a look at the problem if I gave them root on the
box. “Over my dead body,” I thought_

That seems a little silly to me. If they control the hardware, they basically
have root already. They shouldn't _need_ root to investigate the alleged
problem, of course, but the level of trust you need to grant your VPS provider
is essentially the same as giving them root.

~~~
aaronsw
Technically the police can break down my door too, but that doesn't mean that
I'm going to trust them with a key. There are both social reasons (a key is an
invitation) and technical (it's much more noticeable if they break down the
door).

~~~
brk
Yes, but in this case they weren't breaking down your door (at least not yet
anyway). They were asking for a key for entry in order to assist you.

You could have most likely also given them an account and appropriate sudoer
rights.

~~~
aaronsw
Did you read the article? They edited my partition to add their key to my
authorized_keys, logged in with it, and turned off my webserver. Whatever that
is, it's not asking to assist me.

~~~
sho
You make it sound more difficult than it is. In other words, they mounted your
partition, cat mykey.txt > authorized_keys, then ssh'd in and shut down your
misbehaving app. They probably have a script for it since I bet it happens all
the time.

You're dangerously close to whining, did you know.

~~~
_pius
_You make it sound more difficult than it is._

Hmm. I'm sorry, but I don't see how this matters at all Sho. Do the ethics
change based on whether you have a rootkit or you do everything by hand?

~~~
sho
Ethics? Ethics don't enter into it, this is pure business necessity. Fact is,
a $20/month VPS account has no rights and if it causes trouble it will be shut
down, simple as that.

Pay $250/month for an account at Rackspace (or whoever) and you will be
treated very differently.

~~~
SwellJoe
_Pay $250/month for an account at Rackspace (or whoever) and you will be
treated very differently._

Actually, the reason you pay $250 (more, probably) at Rackspace, is _because_
they will login to your box on a regular basis to "manage" things. Rackspace
is focused on "managed hosting"; they help you administer your system, which
is why people pay a big premium for it. Most hosting customers have no idea
what they're doing, and they need a lot of hand-holding, and a "grownup" to
make sure things stay sane on their systems.

~~~
sho
True. A rackspace account is more like $450 for a single server. They keep
their prices secret for a reason.

------
mark_l_watson
I have been using RimuHosting for my own projects and for customers for test
deployments for years - so far I have had good experiences with them. They did
complain once that I was using too much CPU, but I checked, and found problems
(my fault) with a Merb deployment).

------
shabda
In a way, isn't this a reason to use Rimuhosting. They obviously care about
their customers enough, to proactively monitor and step in that everyone gets
a fair CPU share.

~~~
aaronsw
As I note in the article, they already had placed a CPU cap on me. They could
have lowered it to whatever my fair share is instead of breaking into my box
and turning off my webserver.

~~~
blasdel
It would have made a lot more sense for them to just pause your VPS.

------
blhack
Would you mind posting what the offending CGI was doing? If you coded up your
index.cgi to walk all over the CPU as badly as it sounds like you did,
something was probably very _very_ wrong.

That said, they were absolutely in the right to do this (except that they
should have notified you first...).

I saw that you (or somebody) used the analogy of a landlord and an apartment.
Let's explore that further...

Say you are renting an apartment, and the landlord doesn't charge you anything
for water (which is pretty common, if not standard). Now, there is a bit of a
gentlemen's agreement here. Yes, you have "unlimited" water but, like
everything else in life, this needs to have the words "within reason" appended
to it.

Now lets say that, for some reason, your apartment was using 10x as much water
as everybody else in the building...so much so, in fact, that nobody else in
the building could reliably get _any_ water.

Your landlord calls you and says "Hey, joao, would you mind cutting the water
back, please? You're kindof using too much", which you ignore.

A couple of months later, again, the landlord calls

"Hey, Joao, you're still using a shit-load of water, and your sortof hogging
it all. Some of the other tenants are complaining...if you want, leave a key
to your place in the office and we'll go check it out!"

Which you respond "over my dead body".

So, finally, after months and months of you hogging the water, they just use
the master key to the building to go into your apartment, where they see that
you've had the shower, the sink, and the toilet all running _constantly_ for
the last 4 months. They shut them off, then call you

"Hey, joao, yeah, we noticed that you pretty royally messed up and kindof left
EVERYTHING in the bathroom on at the same time. Not cool...so we shut it off.
Sorry for the inconveinience. BTW, sorry for having to go into your appt. like
that, but the other tenants were complaining. Here are some instructions so
that you can change the lock on your door an keep us from being able to help
you like this in the future. Have a nice day!"

And your response to ignoring their totally legitimate requests for you to
stop hogging all the resources is "WHAT A BUNCH OF PRICKS!!"

I think somebody needs to calm down and take a step back from the situation.

~~~
pyre
Great analogy, but you missed the point. (Great analogy because I can still
use it to demonstrate the point) I would be pretty pissed if my landlord use
the master key to access my apartment _when he had a valve in the utility
closet to just shut-off all water to my apartment_.

These are VPSes here. They can just pause the entire VPS instance, or further
cap the CPU usage from what they had already capped it at.

~~~
blhack
Hooray for this analogy! Sadly, you didn't really use it correctly :(.

Pausing the VPS instance would have been like locking them out of the
apartment completely...

I would much rather have a landlord come into my apartment to turn my faucet
off (keep in mind: after I ignored her requests to for MONTHS) than lock me
out of it completely.

~~~
pyre
You missed the 'or further limit his CPU share' part. The entire point was
that they had options to limit his affect on other users without directly
accessing his instance.

In this case, they also have given the users the option of the 'master key.'
Their ssh key is in the authorized_keys file by default, but the users are
allowed to removed it. To me that is a statement to Rimuhosting that this
customer would prefer that they did not access his VPS directly.

------
robertduncan
Am I the only one thinking that Rimuhosting is a rather unfortunate name?

~~~
helveticaman
Fanatical service, man.

------
callmeed
Couldn't agree more. I got a VPS from them after asking HN for VPS
recommendations in Europe.

Had issues from the start. The setup wasn't what I ordered. Once it was setup
properly, I started getting alerts from their staff that my VPS's resources
were spiking– _but this was before I had even logged into the system and done
anything_. That didn't do much for my confidence in their operation.

Anyway, I canceled with them and will probably use Rackspace as they have
UK/European data centers.

------
tszyn
I signed up with RimuHosting a few months ago, since everyone was raving about
their competent and responsive support. So far it has been a mixed bag for me.
They offered to set up some services for me when I was signing up (a
monitoring service and a pop3 server). Whoever was setting these up did a
horrible job; I'm talking about basic mistakes in the config files that
prevented these services from even working. So I would not trust RimuHosting
to set up anything on my server.

On the other hand, when I experienced weird routing problems between my
machine and my home, Rimu eventually offered to move me to a more expensive
London VPS at the same price, even though the dropped packets were not their
fault. So they are willing to go out of their way to make a customer happy.

As I said, a mixed bag. I'm not planning on changing hosts anytime soon.

------
mattculbreth
I've used Future Hosting <http://www.futurehosting.com/> for a few years now,
with two different VPS instances for $30/month each. Never had a problem, and
if they did something like this I'd be shocked.

------
pbhjpbhj
It can't be left unsaid: I assumed (no pun intended) this was a joke posting
when I first saw it, seriously "Rim You Hosting"??!

------
mleonhard
I've had good service from Linode for nearly 2 years:
[http://www.linode.com/?r=73dc8f5748a14c821e64febc5e461a606f6...](http://www.linode.com/?r=73dc8f5748a14c821e64febc5e461a606f6bf7b1)

~~~
chollida1
was the referral link really necessary?

