

Cloud / VPS Apache Performance Comparison - nicklongo
http://chadkeck.com/2009/12/cloud-vps-apache-performance-comparison/

======
iheartmemcache
Full disclosure: Judging by this man's twitter (@ckeck), he seems to be
employed by Rackspace. That may or may not have something to do with the tests
he ran and the settings he chose. (There are an awful lot of "Test failed"
entries under competitor columns, hmm...)

~~~
ckeck
I make no attempt to mask that I am a Rackspace employee and I even updated
the post with a large disclaimer at the top just in case.

The fact of the matter is that the small instances from Linode and Media
Temple could not perform the tests as conducted with their out of box
configurations. I made no enhancements to any platform. Some helpful readers
have pointed out some settings I can change that will allow the benchmarks to
complete successfully so I am going to do a follow-up post with all of these
"tweaks" implemented across all the providers. I am being as objective as
possible here, don't let the fact that I work for Rackspace fog the air.

These tests are also very easy to duplicate on your own if you really want to
test the data set and results.

-Chad

~~~
moe
_don't let the fact that I work for Rackspace fog the air._

If you don't want us to recognize this as the cheap PR-piece it is then
perhaps next time refrain from conclusions like the following (verbatim
quote):

 _The tests show some significant performance benefits when running on the
Rackspace Cloud Servers platform most likely due to (in my opinion) a more
robust network infrastructure, lower hardware utilization, and better host
server hardware._

And from unverified claims like (quote):

 _Even if different web server software is used, it is unlikely that the
results and trends would differ._

Otherwise, if that truly was supposed to be a honest benchmark then please
hand in your engineer-badge at the door...

------
rbranson
I'm about done with people comparing benchmarks on burstable CPU environments
like Rackspace Cloud or Slicehost with a fixed CPU environment like EC2. Over
the ridiculously short period of your benchmark, the burstable environment
will almost always be faster. Over the long term, not so much. Please compare
apples to apples.

~~~
justinsb
Didn't EC2 sell the cloud on the idea of being able to burst? Isn't it better
to be able to do that for short periods without having to fire up a new
server, configure it, and pay for a whole hour?

Your point is very valid from the point of view of benchmarking, but in the
real world, I see bursting as a massive advantage.

It feels like we need a better benchmark suite; like I/O test suites that can
measure the burst rate and the sustained rate.

~~~
rbranson
EC2 scales across instances, it does not burst CPU resources in a single
instance. You can't depend on the "burstable CPU" to be available when you
need it. It's a highly unreliable form of compute time. You just have to pray
and hope no one else is using the same resources on the same physical machine,
or you're SOL.

~~~
justinsb
Right, I agree that scaling out is great. However, bursting is also a great
feature; most loads are bursty (you're caching data and have a cache miss or
timeout, you request an admin page which does a lot of computation.) All time-
sharing involves statistical reasoning, even within your own application: all
requests might be served from cache, or ten people might issue different
cache-miss requests near-simultaneously. You just reason that under normal
circumstances e.g. 80% of your requests will be served from cache, and that
determines your typical load. Short-term bursting means that maybe you can
size your machine closer to the mean load, rather than the 2 or 3 sigma load.

So, I want _both_ bursting and scale-out :-)

The benchmark is fair for measuring CPU bursting performance, but we do need a
more comprehensive benchmark suite.

~~~
moe
_Short-term bursting means that maybe you can size your machine closer to the
mean load, rather than the 2 or 3 sigma load._

No, it doesn't mean that. Because you can't rely on the resources to be
available when you need them. The potential headroom may catch your first two
spikes - and then crumble on the third.

Would you buy a second-hand airbag for your car that only "usually works" but
is a bit cheaper? Thought so.

------
ubercore
Saying "I didn't want to customize the environment" is disingenuous -- it
means you're just benchmarking the default configurations, not the performance
of the host itself. Any benchmark with that many "test failed" results isn't
very valid, or helpful.

~~~
justinsb
I think the idea is to get away from the messy situation you see in TPC
database benchmarks, where the environments are highly tweaked by the vendor
just for that TPC test, and this bears no relation to the performance you
actually get as a real world customer.

I think he's re-running the tests anyway where the vendor suggested a tweak; I
would take a more hard-line approach (if it isn't what you ship, it's not what
I'll test). If the vendor's customer support is responsive and suggests a
reasonable tweak, I can see the need for a more nuanced approach!

~~~
ckeck
Correct, I did not want to have any influence over the results of the "out of
box" testing as many users on these platforms won't even know how to tweak the
OS and Apache on a low level.

But as mentioned, I will be making some of the suggested tweaks to ALL of the
platforms and rerunning the tests and posting a follow-up on that as well. I
will make sure everything is as uniform as possible.

If there is anything specific I should be aware of please let me know, thanks!

-Chad

