

Rackspace vs. EC2: Performance Analysis - timf
http://www.thebitsource.com/2010/01/11/rackspace-cloud-servers-versus-amazon-ec2-performance-analysis/

======
bonsaitree
I was wondering why the text read a great deal like a VNR (video new release).

The analysis may be "independent", but the fine print at the bottom of the
article indicates that it was paid for by Rackspace.

~~~
figured
Fine print: "The Bitsource conducted an independent, third-party performance
analysis of Rackspace CloudServers and Amazon EC2 on behalf of Rackspace Inc.
The results were not influenced in any way by Rackspace Inc. as presented
using the methods in this analysis."

~~~
moe
_The results were not influenced in any way by Rackspace Inc._

Yea. Right.

Rackspace just loves to pay for random pseudo-benchmarks. It's their way to
give something back to the community. Or something.

------
allertonm
Compiling the linux kernel seems like a poor choice of test, since it is not a
pure CPU activity and will perform a significant amount of I/O.

In addition, it doesn't seem like compiling the linux kernel requires very
much RAM, so you get the most bang for your buck using a CS VM with only 256MB
and the cost only goes up from there as you add more RAM that the test cannot
benefit from.

(Edit: I would love whoever downmodded me to explain why I am wrong.)

~~~
pquerna
its a decent enough test, mostly because most geeks out there have done it
themselves, so you can get a good 'feel' for a machine by knowing how long it
takes to compile the kernel.

its not perfect, and yeah, its kinda a geeky-benchmark, but it doesn't make it
an invalid benchmark, as long as it was conducted correctly.

~~~
allertonm
I consider myself a geek (and most of my friends would agree) but it's been 15
years since I last compiled the Linux kernel :)

That said, I wouldn't rule it out as a test because of that, but I do think
that if you are going to compare 4 different RAM sizes of CS instances, and
compare the costs of doing a job using those different sizes then you need to
pick a job that is going to make use of the RAM. You can see from the results
that there is a small (time) benefit moving from 256 to 512MB (but not enough
to offset the increased costs) and virtually none above that.

------
ajross
Crank curmudgeon review: Loses big points for including illegibly down-
filtered graphs in the page. Either make them readable or make them links.
Using fancy animations isn't an excuse for including 600x266 pixel unreadable
blotches in your document.

~~~
SamAtt
All the graphs have links to full size versions and the size in the article
itself is dictated by the site (since the side bar is clearly mandatory) So I
don't quite no what your problem is.

~~~
ajross
I can't read them. What's the point to putting them there if I can't read
them? Just include a link saying "Graph...". Why format a graph to be 1200
pixels wide, with small fonts even at that size, and then shrink it by a
factor of two?

I mean, the inline images might as well be pictures of puppies for all the
value they provide to the page.

Edit: I should add that if the size of the images is dictated by the site,
then some attention might have been provided to including inline images
designed to be used at that size. It's not like it's impossible to display
this information in an 600-pixel-wide format...

------
kylec
It seems from the article that they only tested local disks on EC2, not EBS.
It would be nice to get a comparison with EBS included.

~~~
bonsaitree
Agreed. EBS is really where the rubber meets the road in scaling web apps for
EC2--esp. a de-normalized datastore.

~~~
moe
Amen.

Rackspace knows that very well, wonder why they didn't tell the author to use
EBS for the comparison. Oh wait, I might have an idea...

Anyways, as usual, don't trust any benchmark that you haven't faked yourself.
This one, just like all other rackspace/ec2 comparisons that you see cropping
up these days, was paid for by Rackspace and obviously tweaked to produce
their desired outcome.

Ironically they even point out the flaw in their setup by showing how the CPUs
on the EC2 instances are not fully utilized.

Guess what that means? It means the test was I/O bound.

I'd be curious to see the test repeated, but this time with a RAID10 over a
couple EBS volumes. Not sure Rackspace will pay for _that_ one, though...

------
dgreensp
In my own informal experience, running identical CPU-bound tasks on Rackspace
and EC2, it's just a lot easier to get good performance on Rackspace.

Instance size on Rackspace does affect performance (as their FAQ explains),
but it affects minimum guaranteed performance; there's bursting up to the
capacity of the entire machine you have a slice of. I know there's something
unsettling about relying on excess capacity, but in practice any Rackspace
instance will tend to beat any but the most expensive EC2 instance in terms of
raw CPU.

The sweet spot of EC2 really is offline processing where hundreds or thousands
of machines are spun up by software. If you want to manually fire up a small
but flexible number of fast Linux machines, I would recommend Rackspace.

------
rmason
Who makes a living compiling a Linux kernel? Perhaps web comparison benchmarks
weren't selected because they weren't favorable to Rackspace?

Do they feel their cloud offerings are inferior to Amazon's?

------
carson
This is probably a little better comparison of CPU performance between the
difference platforms and a couple others: <http://journal.uggedal.com/vps-
performance-comparison>

------
travisp
It's nice to see an analysis that looks at more than just CPU. Disk I/O can be
very important in some cases.

It's also nice to see the tests being done over a period of time, rather than
a single day.

