

Performance comparison between EC2, Slicehost, Linode, Rackspace Cloud, Prgmr - uggedal
http://journal.uggedal.com/vps-performance-comparison

======
bdr
Watch out for the bad Y-axis labels on this post: they go from min to max
instead of 0 to max. This doesn't change the ranking, of course, but
exaggerates the relative difference.

~~~
uggedal
Bad form? It was hard to represent the data with a 0 to max Y-axis. Should I
make a note of it?

~~~
bdr
I think so. It's an easy-to-miss detail that really changes your perception of
the data. Ideally the graph would indicate this itself by having a broken
axis, but that's probably a pain to do.

~~~
uggedal
I included a note at the top of the article. The graphs are from Google's API
and were a pain to get in the state they are now. Not going to change them in
this lifetime;-)

~~~
bdr
Yep. For Python there's a good-enough client for the graph API called
"graphy", I think it's on Google Code.

------
MicahWedemeyer
You're in much greater danger of having poor performance and downtime from
your own code rather than anything to do with your host. So, before you jump
ship from one to another based on benchmarks, think about how much time and
effort such a move will take and ask yourself what you could spend it on.

If you're running a web app, my bet is you'll get a much better ROI on time if
you pick an expensive DB query you're running and try to analyze/optimize it.

All that said, if you haven't yet picked a host, this does offer some
interesting insights. Still, I would rank APIs, GUIs, management panels, and
helpful support as way, way more important than raw performance.

~~~
bumblebird
Definitely. Look at the other points as well, like bandwidth.

From what I know linode is also cheapest for bandwidth overage. Far far
cheaper than slicehost.

------
b3b0p
I use prgmr, but it's only personal use right now. I paid the annual for the
1GB plan. The monthly cost is quite low and it gives me my own little server
to deposit all my apps on and what not. I'm very happy with it. The
performance is good enough for me.

I have had no issues at all. Highly satisfied customer here.

~~~
kylec
Did you have any setup problems? I tried Prgmr a few months ago and my
instance wouldn't boot for some reason. Tried emailing the support but didn't
get a response. It was only $6 to give it a shot, but I guess I'm not smart
enough to use it.

~~~
lsc
really? did we at least get you a refund? we do sometimes drop stuff, but we
try to at least refund everyone who we don't get going.

(the vast majority of or problems are provisioning-related, as well... we're
working on that. But still. If you didn't get the refund email me and I'll see
to it.)

~~~
kylec
No contact at all I'm afraid. Don't worry about the refund - I've been
ignoring the emailed invoices ;-) What I'd really like is to give you another
try, so I've sent you an email with the details.

------
cloudhead
Summary: Time to switch to Linode.

~~~
stingraycharles
One thing scares me about Linode: I can not find _any_ information with
regards to availability guarantees at all. Even when they talk about their
"datacenter availability", they're talking about which image slots are
available in which datacenter.

EDIT: Ok, I just found some availability information in their FAQ:

<http://www.linode.com/faq.cfm>

"What we can boast about is our commitment to resolving these issues in the
quickest fashion possible. Most customers will tell you the last time they
rebooted was to take advantage of a plan upgrade. 99.9% uptime, or your lost
time is refunded back to your account. "

This essentially means it's worthless: they're only backing it up with paying
you back the time your server was down (they'd _better_ do that!). Looks like
they don't have much confidence in themselves in that area.

~~~
etherealG
sorry, but I don't think I've ever seen any virtual server hosting provider
that went further than this. most don't even mention paying you back for lost
time, and instead will just give a 99.9% figure without saying what the
rolling time on that % is. To be fair, neither are linode, but at least they
offer your money back, most don't.

~~~
martey
Slicehost's "SLA" is even sillier than the 99.9 percenters:

 _Here’s our SLA: we’ll do our best to keep your machines running smoothly for
as long as possible and get them up ASAP should something go wrong._

~~~
mrkurt
SLA's are pretty silly and useless, as a rule. :) They're really only worth
caring about if they can get you out of some long term contract. They'll never
be good enough to counteract the losses you'll suffer from downtime.

~~~
wisty
SLA's are not useless, they are just useless for commodity hosting.

If (say) the NYSE was negotiating for a host, they would make sure they had an
SLA that gave _massive_ damages if the host failed.

But for commodity hosting, the provider writes the contract. There is no way
they are going to expose themselves to your losses other than a fixed refund
for the time you lost. Not just because of the costs, but because the costs
would be a PITA to calculate. If Twitter goes down and alienates its
customers, how would you calculate the damage?

~~~
mrkurt
Even "SLAs with teeth" are pretty useless, speaking as someone who's been
involved in negotiating them (on both sides). Like I said, the most useful
thing you can get from an SLA is a contract out. Even the NYSE is _never_
going to recoup anything close to their real costs from an outage.

You don't ensure uptime with SLAs, you pick good providers and go for
redundancy.

~~~
ams6110
Additionally such agreements (essentially "insurance" against downtime losses)
are only as good as your willingness to enforce them. Likely you'll be going
to court over anything significant, and paying $100/hr or more to your team of
attorneys.

------
delano
Wow, I'm really impressed with how stable the performance is with EC2. All
straight lines.

------
lsb
To be fair, you can get the small EC2 down to $30/mo if you prepay, and
Slicehost can go down 20% too.

~~~
patio11
Slicehost goes down 20%? I was aware of the 10% payment credit for paying
max(monthly_bill * 6, 240) in advance, which works out to be a 9.1% discount.
Is there an option I'm missing? I wouldn't mind putting a few hundred in my
pocket over the next year.

~~~
lsb
10%, my mistake. Does bingo card creation require a lot of CPU, and, if not,
why not move to EC2? I'm also on Slicehost, looking for better deals.

------
khangtoh
Linode has been awesome since I got a VPS through RailsRumble with them 2
years ago. Till this day, I still use Linode and they have the best price for
VPS as well as features that you get FOC unlike slice host.

And you know what's great about them, they have been quietly supporting the
Rails Rumble community and it's time we show them our support for being best
at what they do!

~~~
cmelbye
Yeah, what they do for the RailsRumble competition is awesome. They set aside
hundreds of instances for competitors during the competition (which is a
wonder on its own), and they give out several large instances as one of the
prizes for the winners.

~~~
kellishaver
I switched from slicehost to Linode after this year's Rails Rumble. It was
seamless and I've had great performance. The one time I had to contact tech
support, my issue was resolved in 7 minutes.

------
oomkiller
My first observation was the standard deviation on some of these measurements,
Zed Shaw would have a heart attack on some of these measurements, and I don't
believe they should be trusted. The best way to measure performance is to run
your app on it, as synthetic benchmarks mean next to nothing compared to your
actual application benchmarks. Finally, if you are this worried about
performance, you should probably be looking at running real hardware anyways,
like GitHub now is.

~~~
npk
The standard deviation is simply a measure of the spread of a distribution.
There's nothing wrong or right about high standard deviations. In fact, the
high standard deviation means that you should expect highly variant
performance.

Look at the figures. The performance of Slicehost follows a sawtooth like
pattern. The quantity standard deviation is useful because it quantifies what
to expect. Plus or minus one standard deviation means that ~ 2/3 of the time
you will fall in that range.

If you think about the problem a little bit, you might be more worried about
the standard deviation of the standard deviation. This, in fact, would be a
useful quantity, but hard to measure.

EDIT below this line ------- Several comments below have commented that SD is
somehow less useful if it's "large" (or large relative to the mean, or
whatever). The reason people think large SDs are indicative of a poor
experiment is that in school lab classes one calculates the SD and call it the
"error".

The standard deviation is a measure of spread, if it's large then the spread
is large. Knowing the spread has value. In this case, under the parent's
experimental conditions EC2's performance is more constant than that of
slicehost's.

A fair critique of the blog posting is that the error on the standard
deviation may be large, depending on the experimental conditions. It is _not_
a fair critique to say that the SD is too high to make a prediction, you just
have larger performance spread. Note that the performance spread described is
not necessarily "error". The spread is inherit to either the server (as
implied by the article) or the method (in which case it is an error).

~~~
oomkiller
I understand the performance variance characteristics, what concerns me about
these tests is that the graphs of them are not continuous, they seem to be
immediate dips instead of gradual curves. This indicates to me that the sample
size wasn't large enough, or at least the graph needs a higher resolution if
data exists to support it. Also, at the bottom, the article gives the
numerical data, with the mean and standard deviation. However, the mean for
the hosts with high standard deviations are essentially useless, because the
standard deviation is so high. If we are going to compare average performance
between cloud hosts, lets at least have useful averages to base our opinion
on.

------
cadwag
I signed up with Slicehost a few months ago. It was my first experience with
VPS hosting and Unix.

As a beginner, Slicehost was really great. The control panel is really
straight forward and easy to use, their support is quick and helpful, and
their how-to articles are really great for beginners.

To quickly compare the how-to's in a similar category:
<http://articles.slicehost.com/ubuntu-hardy> vs
<http://library.linode.com/lamp-guides/>

So they both have articles for the absolute beginners, but it still seems like
Linode slightly favors the more intermediate to advanced users in their how-
to's. Purely subjective, but Slicehost's seem a bit easier to read / follow.

That being said, this article was incredibly interesting. Now that I have a
bit more experience with VPS's and Unix, I am very tempted to head over to
Linode.

I am still developing my web app on Slicehost but I have not seen any
performance problems since I have very little traffic. I am now considering
setting up the production server over at Linode when I'm ready to go live.

Thanks for the great article.

------
rphlx
Somebody please do this for the low-end dedicated providers. You can get Atom
systems for $30-$50/mo now, which should solve the performance variance
(especially on disk IO) that affect cloud/VPSes.

Provide network latency/tput data also, please. :-)

~~~
boundlessdreamz
50$ and you are getting only Atom servers??

Checkout [http://www.hetzner.de/en/hosting/produktmatrix/rootserver-
pr...](http://www.hetzner.de/en/hosting/produktmatrix/rootserver-
produktmatrix-ds/Produktmatrix)

I'm a happy customer

~~~
riobard
The price for dedicated servers in Germany is just crazy ... a friend of mine
is using this one too and I also have an account. If price is a concern (for
testing and personal purposes), find a friend or two to share a baseline
dedicated server is usually much better than using a cheap VPS.

------
docmach
Why does Prgmr offer so much more memory/$ compared to the other providers? Is
there something about the usual workloads that makes RAM less important or is
it based on hardware capabilities?

~~~
stingraycharles
Well, judging by the SLA of Prgmr over here:

<http://book.xen.prgmr.com/mediawiki/index.php/SLA>

it seems like Prmgr is ran by a single person, with a 99.5% uptime guarantee.
I bet this reduces costs quite a little.

~~~
lsc
we're between 2 and 3 people, but yeah, only I'm full-time, and yes, that
reduces costs.

Also, I build my own servers, which saves me some bucks, but it's not orders
of magnitude.

uh, also, compare my SLAs to the other providers before you start saying it's
weak. (I mean, my sla is weak, I agree, but so is everyone else's.)

~~~
mahmud
I have dealt with Nick a few days ago and I found him one of the most
competent, friendliest and patient tech support people I have ever dealt with.
4 hours of his time dealing with my newbie shenanigans when I am only paying
you $20/mo; that alone has won me over and I have been a walking Prgmr PR
agent ever since that day.

~~~
SingAlong
To everyone who doesn't know: #prgmr channel on Freenode

I was once contacted about an old ticket and asked by Will if it was solved,
since it was solved on their IRC channel (and was not marked as solved maybe).
It shows that they make sure they respond to all the tickets. Really sweet of
them to do it.

I agree with mahmud. Same here friend, I've been that "walking PR agent" for
prgmr ever since I started out with them.

Besides Prgmr is for those who like the real raw stuff - bare bones VPS with
shell access.

The SLA is always something you don't want to read with any hosts :P Coz it
includes all the bells and whistles they say they would provide you if they go
down,and that would sound like they would go down :)

There were 3 reasons why I use Prgmr: 1.) They support Paypal (yes, I cannot
use a credit card since I'm not yet eligible to have one in my country) 2.)
The pricing is great 3.) They have this small 128mb pkg on which I can develop
and test my app. So when launching I can upgrade my plan. Thus saving me a few
dollars which might sum up to one more month's payment when I launch :)

------
ryanwaggoner
This is kind of disappointing for fans of Slicehost :( Any reason that we
shouldn't switch?

~~~
patio11
_Any reason that we shouldn't switch?_

I'm not switching because they meet my needs, my customers are happy, and
switching would take me lots of time that I can more profitably use promoting
my software. I don't get paid for doing work, I get paid for solving problems.
Slicehost is not a problem for me.

Your mileage may vary.

~~~
ryanwaggoner
Well put :) I asked the question because server admin is probably the area
where we're weakest and I don't want to waste time trying to solve problems
that could easily be solved by switching to another hosting provider.

------
trunnell
I would like to see a comparison of the APIs provided by these companies.

My understanding is that Amazon is competing on developer and sysadmin
productivity with the depth of their API and focus on manageability-- not on
performance. Is that correct?

~~~
docmach
I think the big advantage of Amazon is that you can scale up and down quickly
so that you aren't paying for services you don't need. They also have the
advantage of better integration with their other services since you don't pay
for transfers within the same region and the transfers are fast since it's all
on their network.

------
wiprosol
I wonder if the Slicehost numbers were generated on an older slice? He did
note that the writeup was six months in the making. If he was on older
Slicehost gear and the latest Linode gear, that would make some difference, at
least it seems like it would.

At one point I had the Slicehost crew set my account to provision slices in
their newest datacenter in Texas. Things seemed snappier after that, though
there could be several reasons why and it's hard to know if it was just a
matter of newer hardware running the Xen dom0.

~~~
uggedal
The Slicehost slice was located at one of the St. Louis data centers.

~~~
wiprosol
On a newly provisioned slice in the Texas datacenter, it would be interesting
to see how the numbers look.

~~~
uggedal
What does cat /proc/cpuinfo give you?

~~~
wiprosol
from my slice in Slicehost's DFW-1 datacenter:

\----

processor : 0 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name :
Quad-Core AMD Opteron(tm) Processor 2350 HE stepping : 3 cpu MHz : 1994.999
cache size : 512 KB fpu : yes fpu_exception : yes cpuid level : 5 wp : yes
flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc
pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch bogomips : 3993.30 TLB size : 1024 4K pages clflush size : 64
cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power
management: ts ttp tm stc 100mhzsteps hwpstate

processor : 1 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name :
Quad-Core AMD Opteron(tm) Processor 2350 HE stepping : 3 cpu MHz : 1994.999
cache size : 512 KB fpu : yes fpu_exception : yes cpuid level : 5 wp : yes
flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc
pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch bogomips : 3993.30 TLB size : 1024 4K pages clflush size : 64
cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power
management: ts ttp tm stc 100mhzsteps hwpstate

processor : 2 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name :
Quad-Core AMD Opteron(tm) Processor 2350 HE stepping : 3 cpu MHz : 1994.999
cache size : 512 KB fpu : yes fpu_exception : yes cpuid level : 5 wp : yes
flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc
pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch bogomips : 3993.30 TLB size : 1024 4K pages clflush size : 64
cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power
management: ts ttp tm stc 100mhzsteps hwpstate

processor : 3 vendor_id : AuthenticAMD cpu family : 16 model : 2 model name :
Quad-Core AMD Opteron(tm) Processor 2350 HE stepping : 3 cpu MHz : 1994.999
cache size : 512 KB fpu : yes fpu_exception : yes cpuid level : 5 wp : yes
flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht
syscall nx mmxext fxsr_opt lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc
pni cx16 popcnt lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse
3dnowprefetch bogomips : 3993.30 TLB size : 1024 4K pages clflush size : 64
cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power
management: ts ttp tm stc 100mhzsteps hwpstate

------
spot
i have two linodes, one for high bandwidth (v2d7c.sheepserver.net) and one for
low (electricsheep.org and scottdraves.com etc). the admin tools are awesome,
there is hardly any (if any) downtime, performance is great, and prices are
low. two thumbs up.

------
jwr
If I remember correctly, Xen will give your instance more CPU if other VMs on
the same physical host do not need it at the moment. If that is indeed the
case, then your CPU performance comparison isn't very useful, as it only shows
that you were lucky to get an otherwise idle machine with some providers,
while others were loaded.

~~~
atambo
It would have probably been better if these tests were run on different host
machines. An easy way to have accomplished this would have been using the
migration feature to switch data centers, etc.

------
johndevor
Before anybody goes and switches hosts, be sure to consider the cost of
actually moving servers. It _does_ take quite a bit of time to move servers,
and while it may be worth it in the long term, it may not benefit very many
people in the immediate future (I'm not saying definitely don't do it).

~~~
coolestuk
If you are running a site that is used by others and you don't have a system
to automatically rebuild your server on another host, I'd say you have bigger
problems than the relative performance of your provider's architecture.

------
lazyant
For people like me who don't need a lot of performance is more important other
issues like support (quality and response time), backups, documentation,
provisioning, uptime etc.

I'd like to see a more complete comparison. I personally use Rimuhosting and
Rackspace, I'm happy with both.

------
pavs
I have used slicehost for a long time it served me well. I don't use them
anymore. One of my favorite option was the ability to increase the size of
slice whenever I needed it and get charged only for the time it was used. Is
it possible to do the same with Linode? As often, quickly and easily as I
want?

If so, I can see myself getting Linode for my next project. Heard a lot of
good things about them in the past.

In defense of slicehost, they have a no-bs interface with good reading
resources to setup machines and very fast and helpful customer service. I was
once more than 10 days late in payment because I had problem transferring my
money to the account I exclusively use only for online payment and they were
very reasonable with me.

~~~
atambo
Linode will charge you at the beginning of your billing cycle and if you ever
resize, which you can do automatically through the control panel, you will
receive a credit for how much you didn't use.

------
jarsj
I wish your study accounted for the elasticity. I end up using 20-30 machines
(sometimes 100) for couple of hours every day. Normally, I would use 3-4 large
instances. I wonder how EC2 compares with others in scenarios similar to
above.

~~~
uggedal
This was a performance comparison (with a price undertone). If you need
elasticity I would just stick with EC2.

------
sh1mmer
I'd also be really interest to see network performance tested too.
Instance->Instance, Cloud colo->cloud colo, cloud->user and cloud->other
clouds/web services.

People are already starting to do the cloud->user
(<http://www.apparentnetworks.com/CPC/scorecard.aspx>) but I think the other
stuff is equally important/interesting.

------
barmstrong
Wow...very disappointing for Slicehost since I'm running most of my stuff on
there. Would love to see what their response to this is (if any).

------
antirez
I'm using a small EC2 instance to run a service that used to run in a very
small real computer. This instance is unbelievable slow. I'll switch ASAP.
Benchmarks are benchmarks but EC2 is slow and overpriced.

------
blackstag
I have a problem going to a service provider who's website doesn't even
load....

<http://www.prgmr.com/>

~~~
johndevor
It loads for me!

~~~
blackstag
odd. I've tried three browsers and nothing and I am having no problems with
other websites... strange.

~~~
lsc
weird. can you ping it?

~~~
blackstag
ping works, but my browsers just say connecting or loading.... forever.

[edit] lol. forever must = 3 minutes in my world... loads afterwards.

~~~
veemjeem
Sounds like a problem with your internet connection :-)

I think too often we lay blame on websites when usually it's a user error.

------
carbocation
This is tremendous. I am considering switching from MediaTemple to VPS.net
right now and I would love to have seen them put to this test!

------
hendler
Did the amazon pricing consider reserved instances? A small instance is $300
for the year - which changes the price/performance ratio.

------
Concours
well, I guess I don't need to try to do anything similar to have the results,
I'm considering a switch to Linode now.

------
dangrossman
This is an apples to oranges comparison. Amazon's "small" instance is far
slower than any other instance type they offer. A medium instance, for
example, costs twice as much but is much more than twice as fast. He also
compared it to all x64 instances; why not use a 64-bit Amazon instance?

If he put an Amazon medium or large instance up against those others, it would
have fared much, much better.

~~~
uggedal
If you read the article I ran the benchmarks on both i686 and x86_64 on Linode
(which gives you a choice). And paying more than $120/month for one instance
is not within my budget for <http://wasitup.com> which I did the comparison
for.

------
iamdanw
Any idea if the amazon one used EBS for the database? Or the new shiny RDS?

~~~
uggedal
I did not use EBS for the database for the same reason I did not configure the
database according to the amount of RAM on each node.

~~~
moe
I'm sorry but this seems to be one of the many flaws in this benchmarks. Kudos
for your efforts but the graphs are almost meaningless, all things considered.

For example the EC2 instance storage is known to be dog slow. I assume that's
the main reason for it looking so bad in your graphs - the picture might
change with EBS.

As others have pointed out, synthetic benchmarks are a tricky beast generally.
For fairness you should have optimized each host to their max potential -
because that's what a regular user would do. But still, the variance in cloud-
hosting makes it difficult to obtain representative figures. Slicehost might
just have looked awesome in a different week...

------
phpappe
The article is a joke. Amazon of course allows 64-bit. In general it all
depends (big time) on the app you benchmark:

<http://php-app-engine.com/static/cloudbench.html>

------
hop
The linode and slicehost affiliate links at the bottom gives this less
credibility.

~~~
pavs
Why? He also has slicehost affiliate link at the bottom.

Edit: I think you should point out, when you edit your original post, where
you only mentioned Linode not slicehost. Thanks.

~~~
theblackbox
Exactly, I would claim the author to be a fool if this wasn't included.... we
are all entrepreneurs here, right?

In the same way I never batted an eyelid with David Welton's Slicehost vs.
Linode[1] article. It was informativve and unbiased and I was happy to provide
some incentive back for that.

[1: <http://journal.dedasys.com/2008/11/24/slicehost-vs-linode> ]

~~~
uggedal
Now, 4 hours in I have 3 Linode referrals. If these stay customers for 3
months I'd get one month of free hosting for <http://wasitup.com> (which I
provide free for all).

