The longer you stay with a provider, the more likely you are to find a tech having a bad day ...
I'm sorry, but the behavior described in the article is completely unacceptable. I don't care how bad of a day anyone's having, you don't break into your customer's server and shut off Apache without notice.
It's not "breaking in". They own the server, and pretty much every host's terms of service allow them to do pretty much anything they deem necessary to maintain service for their other customers, or in response to abuse or law enforcement requests (read the terms of service at your favorite hosting provider; you'll find that you've agreed to this kind of intervention when you signed up).
So, this particular event shows a lack of good judgment on the part of the tech in question, and a lack of common courtesy in not notifying the customer of the issue in advance of taking action...but it is not an illicit act. No "break in" happened here. The options a hosting provider has when things are behaving badly are pretty much all things that a customer will not like. When a dedicated server is compromised and is being used to send spam, for example, the host will simply firewall the whole thing off from the world, shutting down all services. This happens all the time; the effected customer blames the host, sometimes. Sometimes there are other options. In this case, the host opted to turn off one (disputably) offending service rather than kill the whole host. I think we can all agree this wasn't the best way to handle it. Thus my assertion that this is a tech having a bad day (or simply one bad support tech at Rimuhosting, among mostly good ones).
Actually, I think the one thing I'm uncomfortable with is the illusion of propriety (ownership) Rimuhosting has given by saying "You are able to remove that if you are not comfortable with that.", in reference to the ssh key. This is a bit of a disconnect from the reality that under some circumstances, they will have to exert their ownership rights, and there's nothing you can do about it. At some point you just have to trust in the good faith of your provider. But the only alternative to doing what they did (assuming they, in good faith, thought Aaron's VM was damaging service for others), assuming they don't have the ability to throttle Xen instances on this particular machine (maybe it's too old to have that capability, I dunno, or maybe the tech in question didn't actually understand those capabilities; the tech could be new...everybody starts somewhere), would be to turn his VM off completely. From the techs perspective he may have thought he was doing Aaron a favor by leaving mail and other services running.
Again, I'm not arguing it was good judgment on the part of the tech. But to say that Rimuhosting broke into their own machine is ridiculous.
But to say that Rimuhosting broke into their own machine is ridiculous.
I'm not sure about that. Since the VM is virtual, you could say that they don't own the VM, they own the machine the VM runs on. They can control and manipulate the VM via the container for running them (Xen in this case). But since the VM doesn't physically exist, it could be argued that they don't actually own it, they only rent server capacity to run the VM.
Now, I know that you could similarly argue that they were only reading/modifying a disk image that existed on their server, but that isn't as much fun to argue.
I think that everyone here can agree that this was a dumb move by Rimuhosting. And I'm not sure what the contractual arrangement is when you buy a VPS from them... but what they have posted online doesn't make this very clear.
The question of ownership is interesting in this era of cloud computing (and not at all clear). I think it's clear that they own the bare metal. I think it's also clear that you own your data. However, where those mix is an interesting question. At what point is their (uninvited) intervention in a customer's VM warranted outside of their normal management options in the VM container (Xen)? At what point is the intervention too much?
Does the company matter? I can imagine everyone being up in arms if Amazon tried the same thing with an EC2 instance.
Of course this could all be alleviated by using a dedicated server... but even then, if the hosting company owns the hardware, would it be okay for them to pull a hard drive, and add themselves as root? Never. Ever. But this is effectively what we're talking about. The only difference is that with a dedicated server your actions don't affect others. In a VPS environment, your actions can affect others. But I think that there are far better methods for controlling this than "breaking in" to a customer's VM.
CPU is one of the easiest things to throttle in Xen. there's no reason why the provider should care at all that you are running as much cpu as they let you. Proper use of the credit scheduler means this happens without administrator intervention.
You can also gracefully limit disk bandwidth and network bandwidth from the dom0, without ever getting a shell in the DomU.
Yes, and this would have been the correct way to fix the situation. Gracefully reduce the available CPU for the offending VPS, and notify the customer.
Personally, I think freezing the domain is preferable to logging in and killing a (likely rather important) process.
But then, the discussion is academic. I've never had to shut down a domain to prevent heavy legitimate or runaway process use from harming other users.
I have shut down many domains to stop abuse, but that's a very different thing. If you want to examine a compromised disk, you are best off doing so booting off of read-only media then carefully examining the compromised partition, so you want it shut down anyhow.
I think this is actually the heart of the matter. No one (or almost no one) is arguing that a hosting provider shouldn't be able to take steps to prevent one customer from negatively impacting others.
What is at issue is whether the tech should have stopped the instance, or logged in and located and stopped the offending service. The privacy nuts among us (I'm one of them, but with a healthy dose of "hosting industry reality" thrown in for good measure) are crying foul that anyone should ever be able to login to "your" server under any circumstances. The other side of the coin is that a large percentage of hosting customers have no idea what they're doing, and rely on the host to occasionally step in and set things right.
Hosting providers tend to default to stepping in, because that's what the majority of hosting customers want and expect. Rackspace has built a ~$1 billion business on this premise. "Managed hosting" is more expensive because you get this kind of intervention (hopefully in a more positive direction most of the time) on a regular basis.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box. And, since I know Rimuhosting has a good reputation that spans a decade or so, I must assume it was a mistake made in good faith. It is entirely possible the tech didn't reread the full conversation about the problem, and missed the bit about "over my dead body", and just assumed Aaron would want to keep email spinning while he figured out why his website was causing problems.
To turn this into a huge political struggle is pointless, and casts aspersions on a company that is by most accounts, one of the really good ones in their industry.
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box.
The thing is, that's just Aaron that thinks that. I'm sure other customers would be delighted for tech support to access their boxes to fix problems, indeed annoyed if they didn't.
Aaron pays $20 for a cheap-ass VPS, uses too much CPU, and they go out of their way to fix the specific problem instead of just nuking his account, and he complains on some BS privacy grounds. You just can't please some people.
Yeah! You tell 'em! Nobody has any grounds to complain when a specific fucking request is ignored! Who cares if the customer gets what he wants? He should be happy he's getting what thirty other people want instead!
Criminy. Did you even read what you quoted before you responded to it?
You sound like a person at McDonalds. I SPECIFICALLY ASKED FOR A QUARTER POUNDER WITHOUT PICKLES AND YET I FOUND A PICKLE IN MY QUARTER POUNDER AND I WOULD LIKE TO SPEAK TO THE MANAGER
Do you think this is more or less wrong than simply halting the instance?
If they had reason to believe his instance was impacting other users or their network, then halting the instance (preferably after contacting him about the problem, which seems to have happened) would not have been wrong at all. So, logging in after he'd asked them not to is definitely the "more wrong" of the two options. I tend to think it was an honest mistake made by a support tech who was having a bad/busy day rather than an attempt to violate any (real or imagined) privacy pact that the company had made with Aaron.
As I've mentioned a few times here, the default expectation of hosting customers is that the host will login to find and fix problems. From the perspective of most hosting customers, that is part of the service they are paying for (and they pay more to get more of it, as in the case of RackSpace), and if it were missing they would be angry and leave. My guess is that the tech at Rimu simply didn't remember or notice that Aaron had requested no one ever login to his box.
Basically, what I'm trying to say is that, as someone that is deeply familiar with this industry, when I read the shitstorm that has been kicked up here over this (extremely minor) incident, I'm just stunned and baffled. This is seriously a tempest in a teapot, if ever I saw one.
Yes, your hosting provider should respect your privacy, to the best of their ability to do so while shepherding their shared resources responsibly to provide the best service to each of their customers. But, there is no accusation of theft of data; no reason to believe Rimuhosting acted inappropriately with the access they had; no reason to think it was done maliciously or completely without warning (Aaron admits he'd talked with them in the past about the issue); and no reason to think Rimuhosting was trying to piss anyone off.
Not being thoroughly informed about a customer's wishes is pretty much all one can accuse them of here.
I'm sorry, but the behavior described in the article is completely unacceptable. I don't care how bad of a day anyone's having, you don't break into your customer's server and shut off Apache without notice.