

Concurrency's shysters - joubert
http://dtrace.org/blogs/bmc/2008/11/03/concurrencys-shysters/

======
rosejn
I found it interesting that in the post he criticizes previous discussion of
lock based concurrency, while not providing anything close to the level of
analysis that he apparently requires of his fellow disputant.

'“Locks are not modular and do not compose, and the association between locks
and data is established mostly by convention.” Again, no data, no qualifiers,
no study, no rationale, no evidence of experience trying to develop such
systems...'

It is well known that lock based concurrent libraries do not compose, and
anyone writing articles about concurrency should not require an explanation of
this central issue in the discussion[1]. Of course with millions of hours of
developer investment in a centrally managed codebase like Solaris it is
possible to achieve incredible parallelism with locks, but that isn't
practical for the vast majority of modern software, which continues to use
ever more external libraries.

Clojure's STM paired with immutable data structures are an especially well
suited combination that are unlike most of the research systems presented in
this area. Of course there are additional resources that will have to be
managed, such as access to I/O, but I would argue that access to I/O is by far
the easier problem to deal with in comparison to concurrent access to memory -
primarily because I/O devices are typically accessed serially so you can just
queue up requests.

[1]: Here's one explanation with a code example of why locks don't compose:
[http://www.drdobbs.com/article/printableArticle.jhtml;jsessi...](http://www.drdobbs.com/article/printableArticle.jhtml;jsessionid=SQVVQ5O0S2ACLQE1GHOSKH4ATMY32JVN?articleId=202802983&dept_url=/cpp/)

------
lukev
He's kind of tilting at windmills, here. It's true that there's a lot of hype
about concurrency, etc.

But take Clojure, which AFAIK is the only real system people are actually
using that places STM front and center (except maybe something in Haskell
land). Rich Hickey is the furthest thing possible from a shyster.

For example, he never claimed that STM is faster than manual locks, just that
it's an order of magnitude easier to write correct programs - which seems to
be borne out in actual experience. Yes, there's a not-insignificant overhead.
But for most programs it's still probably worth it.

Similarly, the claim that STM isn't any good because most computational
resources aren't memory is a complete red herring - STM _is_ quite useful for
memory management, and for other resources, well, find a different strategy.
It doesn't stop STM from being quite useful in its domain.

All in all, this sounds more of a luddite rant than a reasoned opinion. It
reminds me of what I've read of the initial objections to managed memory or
even high-level languages themselves. Of course a lot of the technology isn't
there yet. Of course a lot of it is going to be rabbit trails that ultimately
don't deliver. It still doesn't mean it's not the future.

~~~
cageface
I don't think you can call one of the inventors of D-Trace a luddite, and the
article was hardly a rant.

If you agree that many resources that require serial access are not memory
then I also don't see how STM is making your life significantly easier.
Composing locks _and_ transactions in the same system sounds much harder to me
than just using locks.

I think the jury is still out on the effectiveness of Clojure's concurrency
model. It makes for nice short examples but I'm not aware of anything of any
real complexity built on it yet.

~~~
swannodette
Many of the arguments made here against STM (and in the linked papers) are
nicely addressed by Clojure's _specific implementation_ of STM. This article
is from 2008, before Clojure was more widely known.

More importantly, Clojure is wisely designed with several other _equally_
useful concurrency mechanisms: atoms for serial access, promises for dataflow,
pmap for divide and conquer, and agents for uncoordinated concurrent
activities.

~~~
cageface
How _specifically_ does Clojure's STM implementation avoid the pitfalls
outlined in the ACM paper? How many real-world benchmarks do we have
demonstrating Clojure's concurrency scaling?

~~~
swannodette
An informative and lengthy discussion between two experts (Cliff Click of Azul
Systems and Rich Hickey): [http://www.azulsystems.com/blog/cliff-
click/2008-05-27-cloju...](http://www.azulsystems.com/blog/cliff-
click/2008-05-27-clojure-stms-vs-locks)

~~~
cageface
An interesting discussion but inconclusive. I agree with Rich that it would be
very useful to have an established set of meaningful concurrency benchmarks.

For example, I'd like to see if a Clojure solution of the benchmarks in the
linked ACM paper can scale better than the solutions tested in the paper.

~~~
swannodette
Except those ACM microbenchmarks are exactly the kind of fine-grained problems
Rich Hickey specifically stated he's not trying to solve.

For example [http://dosync.posterous.com/stm-couchdb-and-
pushing-5500-doc...](http://dosync.posterous.com/stm-couchdb-and-
pushing-5500-documentssecond). I added only 7 lines of code. Raw Aleph runs at
about 20k req/s on an 8 core box. This code performs at about 14-18k req/s. So
clearly some contention is at play. I could write this with locks, but I have
better ways to spend my time. That's the real cost-to-benefit ratio.

~~~
cageface
Stuffing a bunch of things into a DB in parallel doesn't seem like a very
interesting problem to solve to me. As discussed earlier, Clojure's
concurrency primitives are only useful for a somewhat specific class of
problems: computationally intensive tasks in a single shared memory space. The
benchmarks in the ACM paper seem to me to have a lot more to do with the kinds
of complex datastructures you're going to employ in those scenarios than
batching writes to a key-value store.

------
noelwelsh
I disagree. I think the author is unjustifiably extrapolating conclusions from
a C/kernel development world to the rest of software development. For example,
a bunch of systems use the "green threads" model. Erlang SMP is one. Erlang
SMP is unarguably successful. The referenced article on STM confuses
parallelism with concurrency[1]. They test on a bunch of benchmarks that are
mostly 1) not concurrent and 2) for performance rely on fast numeric
operations that have a vastly different level of granularity to typical
business ops.

[1] Parallelism = speeding things up by using multiple cores. Concurrency =
modelling a system that naturally has different activities going on at once.
These are not the same thing.

~~~
kscaldef
Erlang SMP is successful on one axis (ease of writing correct and reliable
concurrent problems), but it seems to me that the argument in the article was
about the performance costs of the model. And, at least as of a year or two
ago, Erlang SMP did _not_ have good performance. There was a minimum factor of
2 overhead in total CPU usage just by turning on SMP, and heavily concurrent
programs could run massively slower
([http://shootout.alioth.debian.org/u64q/benchmark.php?test=ch...](http://shootout.alioth.debian.org/u64q/benchmark.php?test=chameneosredux&lang=all)).
At least part of the problem is that there's no affinity between Erlang
processes and scheduler threads (and, therefore, CPU/cores), so you've got
constant cache misses. There's probably other issues as well. As the author
says, it's a hard problem.

~~~
igouy
> There was a minimum factor of 2 overhead in total CPU usage just by turning
> on SMP

n-body without SMP 123.30s

n-body without SMP 123.30s elapsed

n-body with SMP 133.52s

n-body with SMP 133.57s elapsed

spectral-norm without SMP 46.68s

spectral-norm without SMP 46.68s elapsed

spectral-norm with SMP 47.71s

spectral-norm with SMP 13.90s elapsed

mandelbrot without SMP 230.07s

mandelbrot without SMP 230.10s elapsed

mandelbrot with SMP 253.66s

mandelbrot with SMP 87.76s elapsed

Maybe that Erlang chameneos-redux program is unusual?

~~~
kscaldef
"at least as of a year or two ago". So, maybe, as other people have pointed
out and as I've already acknowledged, the situation may have improved.

