
The Trouble With Multicore - rpledge
http://spectrum.ieee.org/computing/software/the-trouble-with-multicore/0
======
dkersten
Maybe I'm naive, but I don't think the problem is what everyone always says -
that parallel programming is super hard and most programmers cant effectively
program for multicore systems. Instead I think the problem is one of education
and training. How many programmers actually get decent training and education
in multicore programming? I don't think its very many. Of all the programmers
complaining how hard it is to write effective multicore programs, how many
have actually read books and research papers on the subject? Again, I'd wager
not many.

For example, In my four year computer science degree, a lot of time was spent
on high level languages like Java, C, C++, Haskell, even prolog. A good bit of
time was spent on low level details (two computer architecture modules, an
"advanced" computer architecture module, an assembly programming module). Some
time was spent on computability, complexity and parsing. Of course, we had a
number of math modules too, including probability, statistics and linear
algebra. A lot of time was spent on object oriented programming, data
structures and algorithms. A little bit of time was spent on other topics like
network programming, databases, operating systems (we did cover deadlock and
_some_ concurrent programming stuff in great detail here) and AI. The rest was
split between single (optional!) modules on niche areas: digital signal
processing, cryptography, compression, 3d graphics and so on.. including
concurrent programming.

We spent so much time learning how to program sequentially, why would anyone
even suggest that we should be able to program for multicore systems? Were we
given years of training in parallel programming languages, algorithms, data
structures and libraries? Nope. Instead the time was spent on sequential
programming. Of course its going to be hard to program for multicore!

Heres a car analogy:

 _Its like spending years learning to drive tractors, expecting that you can
then drive formula one cars._

Some problems simply don't make much sense to try and solve with parallel
programming. Some problems do, but we're not properly educated to spot these
effectively, to know what data structures are appropriate and what algorithms
we should use, nevermind things like testing and debugging parallel software.

As an aside, if performance is what we're trying to achieve with multicore
programming, then we need to be taught about efficient cache usage too! Misuse
of the processor cache kills multicore performance and popular software
development principles, like object-oriented programming actually works
against the processor cache! Ugh.

There is plenty to help us effectively program for multicore. A short (and
very incomplete) list:

    
    
        - monitor based concurrency (mutexes, semaphores, condition variables + threads); usually not the best option
        - Software Transactional Memory; replace manual synchronisation with transactional access to shared memory
        - Message passing; multiuple threads/tasks communicating through immutable messages (being immutable means that synchronization concerns are minimized)
        - Dataflow languages/libraries where data "flows" through computational nodes and many streams of data can flow through in parallel (each node can be processing at the same time)
        - Tools and libraries such as OpenMP, OpenCL and Intel Threading Building Blocks
        - Languages with a focus on parallel programming like Erlang and Clojure (I especially like Clojures view on *time*)
        - Languages with built-in immutable data structures help make synchronisation less painful. Same goes for atomic data structures.
        - Entity Systems[1] are actually surprisingly suited for multicore programming, IMHO
    

I would suggest everybody to pick up the book _The Art of Multiprocessor
Programming_ [2] and get an introduction to the basic concepts. After that it
depends on what you want to do, but if you're a C++ programmer, I would
suggest the intel threading building blocks book[3].

[1] [http://t-machine.org/index.php/2007/09/03/entity-systems-
are...](http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-
future-of-mmog-development-part-1/)

[2] [http://www.amazon.com/Art-Multiprocessor-Programming-
Maurice...](http://www.amazon.com/Art-Multiprocessor-Programming-Maurice-
Herlihy/dp/0123705916)

[3] [http://www.amazon.com/Intel-Threading-Building-Blocks-
Parall...](http://www.amazon.com/Intel-Threading-Building-Blocks-
Parallelism/dp/0596514808/ref=sr_1_1?ie=UTF8&s=books&qid=1277999747&sr=1-1)

~~~
dkersten
Quote from page 18 of the O'Reilly book on Intel Threading Building Blocks:

 _One of the truths in programming is this: the best serial algorithm is
seldom the best parallel algorithm, and the best parallel algorithm is seldom
the best serial algorithm._

Following on from what I said in my previous post, since we are being educated
in serial algorithms, we cannot realistically expect to write decent parallel
algorithms in a way similar to how we write serial algorithms.

