
On HTTP Load Testing - ctide
http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
======
Locke1689
The author makes this point, but in my opinion not strong enough.

 _While it’s tempting to say that the way to achieve this is to run everything
on VMs, I’m not convinced that adding another layer of abstraction (as well as
more processes running on the host OS) is going to lead to more consistent
results. Because of this, dedicated hardware is best. Failing that, just run
all of the tests you can in one session, and make it clear that comparisons
between different sessions don’t work._

As someone with pretty in-depth knowledge of how a VMM (virtual machine
monitor) works on a very low level, using a VM is a truly awful idea. VM exit
incidence is pretty much completely unpredictable and can have a huge impact
on performance. Not only that, but the behavior of some cases in hardware VMM
layers can cause pretty much complete TLB wipes, thus destroying cache
coherency and any memory optimizations the web server takes advantage of.

Now, if your goal is measuring performance of web servers on VMMs, go ahead,
but be aware that performance consistency is not a plus.

~~~
troels
Did you mean to not use a VM for the test client, or did you mean that testing
performance of a VM is tricky?

~~~
spydum
I'm sure he meant testing performance of a VM is tricky, but in actual fact,
both are true. Using a VM for the test client is just as susceptible to the
issues mentioned as running it as the server. As the article suggests: know
your load generator client. This is difficult if it's not consistent.

------
morganpyne
There are some good nuggets in this article, but he seems to miss a point that
I feel is very important and yet is often overlooked in real-world web load
testing; emulating an actual client behaviour under different real-world
scenarios. Hitting a page or pages X times usually bears almost no resemblance
to how a real-world application is used and therefore gives no indication of
actual performance and capacity.

To test a web app properly you need the ability to script and/or replay real-
world session browsing behaviours, simulating typical interactions. This can
be both very time consuming and difficult to set up but yields far more useful
and realistic results. In the past I have reasonable success using tools like
The Grinder (barely maintained and poorly documented, but very useful and
scriptable in Python), and even Microsoft's Web Application Stress tool
(scriptable in VB). There are probably better tools than these out there but
these at least allow you to code a detailed interaction with you site.

~~~
jacques_chester
> There are some good nuggets in this article, but he seems to miss a point
> that I feel is very important and yet is often overlooked in real-world web
> load testing; emulating an actual client behaviour under different real-
> world scenarios.

I don't think he missed that point; he specifically limits his scope to HTTP
daemons:

> A lot of people seem to be talking about and performing load tests on HTTP
> servers, perhaps because there’s a lot more choice of servers these days.

Testing web apps for performance is, as you point out, more difficult. There
are tools for taking integration tests and running them in large batches. With
some good instrumentation this is a kind of systemic profiling of your app
architecture.

But _all of his points will still apply_.

~~~
morganpyne
Perhaps I didn't phrase my comment as well as I should have. Indeed all his
points apply and are excellent. As you pointed out, his focus is on testing
the HTTP server for performance, rather than the actual apps running on them.
The point I was trying to make is that all of the above, _plus_ additionally
integration testing and app specific testing should really be done when
preparing for the onslaught of a real-world deployment. Admittedly this is
deliberately beyond the scope of his article.

~~~
jacques_chester
Happily, I think we can agree to agree!

