

An update on the Memcached/Redis benchmark - spahl
http://antirez.com/post/update-on-memcached-redis-benchmark.html

======
vosper
I'm enjoying following this back-and-forth between memcached and Redis; I
learn something from each post.

And it's nice to see it being conducted in such a gentlemanly fashion!

------
wglb
The pursuit of benchmarks is fraught with peril. So it is nice to see antirez
taking care to evaluate and thoroughly analyze where the differences are
between two benchmarks.

One of the conclusions that I take away from this is that threading is not a
net win in the cloud. CPUs are so cheap that (to a first-order approximation)
spending development time on threaded programs takes development time away
from developing features.

------
jacquesm
key insight here:

> in a future of cloud computing I want to consider every single core as a
> computer itself.

And that leads directly to a shared nothing architecture, which I think is
definitely an advantage. It's KISS all over, multi-threading is often not the
right decision. It leads to more complex (and sometimes very hard to debug)
software that will have only marginal gain (or even a drop as in this case)
over running multiple well behaved instances of the same process.

~~~
ergo98
_key insight here:

> in a future of cloud computing I want to consider every single core as a
> computer itself._

I didn't interpret that as an insight more than a rather weak attempt at
squeezing some lemonade out of lemons. I also don't see how creating massive
sharding on a single machine in any way corresponds with KISS (the newest
processors have from 8 to 12 cores. 24 cores and up is coming in the very near
future. So on my 48-core 4-processor relatively low end machine I'm to shard
Redis 48 ways? I'm to run 48 instances of Linux? I'm to endure a devastation
of cache coherency?)

~~~
antirez
Ergo we plan to add options so that Redis can do this automatically for you.
You'll be able to say, start N instances. And when clustering will be
available you can configure all this instances with a single common
configuration.

For now this should be done manually, but you have 4 or 8 cores so it's not a
big problem IMHO.

The idea is that there is no difference between the instances you have among
different servers, and the ones you have locally. Given that every instance
can hold a set of unrelated keys, why centralizing it in some way if
completely separated threads can lead to max performances, and simpler code?

Note that 48 independent threads or 48 processes are actually the same thing
more or less, but nothing prevents us from starting N threads at startup if
needed for some reason in the future, each with completely separated state.

I don't understand your parallel about starting N operating systems btw...

Anyway, this is not a problem just with Redis. Programming languages like Ruby
are a mess with the usual conception of threads. And most Ruby programmers
also think far too high level to really deal with a low level stuff such as
synchronization. A shared nothing model with message passing, and one
interpreter per thread, could lead to the maximum benefit while retaining the
more abstract interface to the programmer. Believe it or not the time for old-
fashion multi thread programming is gone (I hope).

~~~
shin_lao
48 independent threads and 48 processes can be very different on some
operating systems.

A process is more heavyweight.

You will - amongst other things - increase the charge in terms of handles. For
example, 48 processes maintaining the same network connections is a waste of
resources.

I'm afraid you might hit a wall with your single threaded approach.

This is definitely not the road we have chosen (but we wish the best of luck
nonetheless! :) ).

~~~
antirez
In a shared nothing environment you need 48 different sockets, distinct data
structures, and so forth. Given the copy-on-write strategy of many OSes the
overhead can really be minimal (you'll not get N copies of the executable in
memory). The context switching should be more or less the same cost... what I
mean is that it's trivial to turn the N proesses design into N independent
threads design, as there is no problem with locking, but it is probably not a
win.

