

The benchmark you’re reading is probably wrong - coffeemug
http://www.rethinkdb.com/blog/2010/07/the-benchmark-youre-reading-is-probably-wrong/

======
antirez
Another common flaw in broken benchmarks: use a single client instead of many
simultaneous clients. In this configuration you are mostly measuring the RTT
plus the client lib implementation latency.

Another one is testing on-disk DBs with datasets that fit in RAM: the whole DB
data will be cached in OS buffers and you don't get the picture of how the DB
will perform with big data.

Benchmarking is one of the most complex tasks, but for some reason
benchmarking is usually not done by the most expert guys in the field.

------
mfukar
It's been a long time since I've benchmarked a RDBMS, but it's always the same
story: Benchmarks are rarely done correctly. People think a benchmark as the
definitive way to prove their point, so they hastily and sloppily design a
simplistic experiment to measure their desired quantities and then cite it all
over the place as _The Truth_.

For almost 8 years I've had to constantly scrutinize people's benchmarks
(worked in networking and distributed systems research, and measurements are
crucial in those contexts), point out their errors, and make them think about
the details and "hidden" factors that affect their experiments. I haven't met
more than a handful of people that could get a benchmark right on their own,
and none (including myself) that get it right the first time.

It's good that there are people that could point our errors out.

------
nuxi
Ironically, this blog wasn't benchmarked either: 504 Gateway Time-out
nginx/0.6.35 The main page works though.

~~~
bradleyland
Given that it's a 504 error, it looks like the app server died or is out of
threads/processes. It's hard to take technical advice from someone who doesn't
do simple things like enable caching in their CMS. Chances are nginx would
still be serving the content if it were a cached resource.

~~~
mahmud
You would be wrong to dismiss the blog post and its author.

~~~
bradleyland
I cannot dismiss what I cannot access.

~~~
mahmud
The front page still works, go there and look around. FWIW, this fancy ginger
cat is their CTO:

<http://www.defmacro.org/>

And I have it on good authority that he knows a thing or two about tweaking a
"CMS" ;-)

