
Ask HN: On Rob Pike's Concurrency is not Parallelism? - lazydon
This is regarding slides by Rob Pike with above title. Every time I go thru this I feel like a moron. I'm not able to to figure out the gist of it. It's well understood that concurrency is decomposition of a complex problem into smaller components. If you cannot correctly divide something into smaller parts, it's hard to solve it using concurrency.<p>But there isn't much detail in slides on how to get parallelism once you've achieved concurrency. In the Lesson slide (num 52), he says Concurrency - "Maybe Even Parallel". But the question is - When and How can concurrency correctly and efficiently lead to Parallelism?<p>My guess is that, under the hood Rob's pointing out that developers should work at the level of concurrency - and parallelism should be language's/vm's concern (gomaxprocs?). Just care about intelligent decomposition into smaller units, concerned only about correct concurrency - parallelism will be take care by the "system".<p>Please shed some light.<p>Slides: http://concur.rspace.googlecode.com/hg/talk/concur.html#title-slide
HN Discussion: http://news.ycombinator.com/item?id=3837147
======
dangets
I believe the concept says to focus more on task-based parallelism rather than
data-based parallelism.

In Go, it is easy to create multiple tasks/workers each with a different job.
This is implicitly parallelizable - each task can (but doesn't have to) work
within their own thread. The only time when the workers can't run in parallel
is when they are waiting on communication from another worker or outside
process.

This is opposed to data level parallelism where each thread is doing the
nearly exactly same instructions on different input, with little to no
communication between the threads. An example would be to increase the blue
level on each pixel in an image. Each pixel can be operated on individually
and be performed in parallel.

So - the push is for more task-based parallelism in programs. It is very
flexible in that it can run actually in parallel or sequentially and it won't
matter on the outcome of the program.

~~~
pcwalton
I don't agree with that; task parallelism is easier than data parallelism in
an actor/CSP-based system, but both have their place. Take something like x264
-- task parallelism will not help it, unless you're encoding multiple videos
at one time. But data parallelism (SIMD, in particular) is the reason it's the
fastest encoder around.

~~~
KirinDave
In general, task paralellism can model anything data parallelism can and given
"a sufficiently smart compiler" you can end up with the same result. This is
an informal corollary of Needham's Duality (which is itself informal, so make
of it what you will).

Our current hardware tends to offer great data parallelism for homogenous task
queues and task parallelism for heterogenous task queues. Given that, we task
parallelism needs a lot more consideration from a human. It's also the case
that our current implementations of data parallelism tend to focus on shared
memory computations, so their scope is a lot more limited than the
distributed-system-conflated discipline of task-oriented concurrency, where
we're currently having an explosion of engineering.

~~~
pcwalton
For a "sufficiently smart compiler" you mean autovectorization. That's a
_hard_ problem.

I've mentioned before that I don't think SIMD is going away anytime soon. It
has so many upsides (cache locality, simple implementation in hardware due to
the single-instruction nature of it) that I think designs that don't take
advantage of it will always be at a disadvantage for the foreseeable future.

------
arebop
I think his point is that although people are often interested in parallel
behavior, they should really focus more on concurrent design to avoid/remove
the dependencies that ultimately limit parallelism. Slide 19 mentions
automatic parallelization, but his point is that developers should think more
about concurrency and not that Go will automatically maximally parallelize
concurrent programs.

~~~
rvirding
Yes. For the underlying system to be able to detect and run in parallel you
need the concurrency. Without concurrency your application will be run
sequentially.

------
aphyr
Concurrency is more than decomposition, and more subtle than "different pieces
running simultaneously." It's actually about _causality_.

Two operations are _concurrent_ if they have no causal dependency between
them.

That's it, really. _f(a)_ and _g(b)_ are concurrent so long as _a_ does not
depend on _g_ and _b_ does not depend on _f_. If you've seen special
relativity before, think of "concurrency" as meaning "spacelike"--events which
can share no information with each other save a common past.

The concurrency invariant allows a compiler/interpreter/cpu/etc to make
certain transformations of a program. For instance, it can take code like

    
    
        x = f(a)
        y = g(b)
    

and generate

    
    
        y = g(b)
        x = f(a)
    

... perhaps because b becomes available before a does. Both programs will
produce identical functional results. Side effects like IO and queue
operations could strictly speaking be said to violate concurrency, but in
practice these kinds of reorderings are considered to be acceptable. Some
compilers can use concurrency invariants to parallelize operations on a single
chip by taking advantage of, say, SIMD instructions or vector operations:

    
    
        PIPELINE1  PIPELINE2
        x = f(a)   y = g(b)
    

Or more often:

    
    
        [x1, x2, x3, x4] = [f(a1), f(a2), f(a3), f(a4)]
    

where f could be something like "multiply by 2".

Concurrency allows for cooperative-multitasking optimizations. Unix processes
are typically concurrent with each other, allowing the kernel to schedule them
freely on the CPU. It also allows thread, CPU, and machine-level parallelism:
executing non-dependent instructions in multiple places at the same wall-clock
time.

    
    
          CPU1        CPU2
        x = f(a)    y = g(b)
    

In practice, languages provide a range of constructs for implicit and explicit
concurrency (with the aim of parallelism), ranging from compiler optimizations
that turn for loops into vector instructions, push matrix operations onto the
GPU and so on; to things like Thread.new, Erlang processes, coroutines,
futures, agents, actors, distributed mapreduce, etc. Many times the language
and kernel cooperate to give you different kinds of parallelism for the same
logical concurrency: say, executing four threads out of 16 simultaneously
because that's how many CPUs you have.

What does this mean in practice? It means that the fewer causal dependencies
between parts of your program, the more freely you, the library, the language,
and the CPU can rearrange instructions to improve throughput, latency, etc. If
you build your program out of small components that have well-described inputs
and outputs, control the use of mutable shared variables, and use the right
synchronization primitives for the job (shared memory, compare-and-set,
concurrent collections, message queues, STM, etc.), your code can go faster.

Hope this helps. :)

------
thebigshane
I don't think he's saying that you shouldn't concern yourself at all with
parallelism; only that you should focus on concurrency first and that will
lead to easier parallelism. And he I think he is saying that decomposition and
concurrency helps non-parallel programs stay simple and easy to understand.
The benefits of concurrency are greater than just parallelism.

I think that is about what you are saying.

------
jberryman
Obviously both terms get used in a variety of overlapping ways. Without
looking at how the terms are used in the slides you refer to, I think the
proper definitions are:

Concurrency is a property of a program's semantics, usually seen in a 'thread'
abstraction. The most important part of concurrency is nondeterminism.
Concurrency might permit parallelism depending on hardware, language runtime,
OS, etc.

Parallelism is a property of program _execution_ and means multiple operations
happening at once, in order to speed up execution. A program written to take
advantage of parallelism can be deterministic, but often is accomplished by
way of concurrency in OS threads. Because most languages still suck.

------
Xantix
The difference between Concurrency and Parallel is in my opinion somewhat
subjective, depending on how you view the problem.

Examples of Concurrency:

1\. I surf the web And I run an installer for another program.

2\. One gopher brings empty carts back, while another brings full carts to the
incinerator.

The idea of concurrency is that two completely separate tasks are being done
at the same time. There may be synchronization points between the two tasks,
but the tasks themselves are dissimilar.

Viewed in one way moving empty wheelbarrows may be completely different from
moving filled ones.

Viewed in another way, they might seem very similar.

Concurrency has to do with task parallelism.

Parallel has to do with data parallelism.

There's a gray line between the two where you can't clearly differentiate
between them.

------
dkersten
IMO, they solve two different goals. Sometimes these goals overlap, but not
always. Please someone correct me if I'm wrong, but this is how I see it:

The goal of concurrency is to model a problem that is easier or better or more
natural to model _concurrently_ (that is, different parts are running
simultaneously). For example, if you are simulating agents in some virtual
world (eg a game), then it may make sense that these agents are being modeled
in a way that they are all running simultaneously and the processing of one
does not block the processing another. This could be done by timeslicing
available processing between each agent (either by using the processor/OS pre-
emptive multitasking, if available, or through cooperative multitasking[1]),
or is could be done by physically running multiple agents at the same time on
different processors or cores or hardware threads (parallelism). The main
point is that concurrency may be parallel, but does not _have to be_ and the
reason you want concurrency is because it is a good way to model the problem.

The goal of parallelism is to increase performance by running multiple bits of
code in parallel, at the same time. Concurrency only calls for the illusion of
parallelism, but parallelism calls for real actual multiple things running at
the exact same time and so you must have multiple processors or cores or
computers or whatever hardware resources for parallelism, while concurrency
can be simulated on single core systems. Parallel code is concurrent code, but
concurrent code is not necessarily parallel code.

Distributed programming is parallel programming where the code is running in
parallel, but distributed over multiple computers (possibly over the internet
at distant locations) instead of running locally on one multi-core machine or
a HPC cluster.

From stackoverflow[2]:

    
    
        Quoting Sun's Multithreaded Programming Guide:
        
        Parallelism: A condition that arises when at least two threads are executing simultaneously.
        
        Concurrency: A condition that exists when at least two threads are making progress. A more generalized form of parallelism that can include time-slicing as a form of virtual parallelism.
    

As for when can concurrency be turned into parallelism, that depends. Assuming
that the hardware resources exist (or that it simply falls back to time-sliced
concurrency if they do not), parallelism can be achieved if there are multiple
things that can execute independently. There are at least three types of
parallelism and if your problem, code and/or data fit one of these, then your
concurrent code may be executed in parallel.

1\. You have a number of items of data and each one can be processed
independently and at the same time. This is the classic _embarrassingly
parallel_ data parallelism. For example, you have a large array of input data
and an algorithm that needs to run on each one, but they do not interact.
Calculating pixel colours on your screen, or handling HTTP requests, for
example.

2\. You have two or more independent tasks doing different things that are run
in parallel. For example, you have one thread handling the GUI and another
thread handling audio. Both need to run at the same time, but both run
independent of each other with minimal communication (which can happen over a
queue, perhaps).

3\. Sometimes you have a stream of data which must be processed by a number of
tasks one after the other. Each task can be run in parallel so that if you
have a stream of data, item[0], item[1], item[2], etc (where 0 is first in the
stream, 1 is next and so on) and a number of tasks that need to run in order:
A, B, C - then you can run A, B and C in parallel such that A processes
item[0] while B and C are idle, then B processes item[0] an A processes
item[1] and C is idle. Then C processes item[0] while B processes item[1] and
A processes item[2] and so on. This is called pipelining and as you probably
know is a very common technique inside processors.

Of course, all three can be combined.

[1] Could be a coroutine which is yielded or simply by executing some kind of
update function which, by contract, must not block

[2] [http://stackoverflow.com/questions/1050222/concurrency-vs-
pa...](http://stackoverflow.com/questions/1050222/concurrency-vs-parallelism-
what-is-the-difference#1050257)

------
halayli
Take a look at lthread:

<https://github.com/halayli/lthread>

lthread supports concurrency and parallelism using pthreads. Each lthread
scheduler runs its own lthreads concurrently, or better said, one at a time.
But from an observer's perspective they look like they are running in
parallel.

Now if you create 2 pthreads on a 2 core machine and each runs an lthread
scheduler then you have true parallelism because you can have 2 lthreads
running in parallel at the same time. One by each scheduler.

I feel this is a closer context to what Rob is discussing than what I found in
the comments here.

~~~
dkersten
Just noticed the license changed to BSD! I had not realised - will definitely
need to give it another look as I may be able to make use of it now. Awesome.

~~~
halayli
Yup I changed it couple months ago :)

------
Mr_T_
Related to this topic:
[http://existentialtype.wordpress.com/2011/03/17/parallelism-...](http://existentialtype.wordpress.com/2011/03/17/parallelism-
is-not-concurrency/)

------
the1
Parallelism is when you run your program on multiple processors. Semantics of
your program does not change whether you run it on single processor or
multiple processors.

Concurrency is when you write your program using multiple threads. Your
program looks and means vastly different if you use threads.

You use concurrency not for performance gain, but for clarity of your program.
You use parallelism for performance gain, to utilize all your processors.

~~~
gruseom
That can't be right. Threads don't improve the clarity of most programs;
they're notoriously unclear.

~~~
aphyr
On the contrary, I think proper concurrency constructs, including threads, do
improve the clarity of programs. Part of the problem in reasoning about
threads is a lack of useful primitives. Multithreaded programming in Java? For
me, at least, it's _tough_. In Erlang? Trivial. In Clojure, if you're willing
to deal with the slowness of the STM, it can be _beautifully_ simple.

~~~
gruseom
Erlang and Clojure don't have threads in the common meaning of the term
(<http://en.wikipedia.org/wiki/Thread_(computing)>). They are reactions
_against_ programming with threads.

You can expand the term "thread" to mean "concurrency in general" but even
then it isn't true that the main purpose of writing concurrent code is
clarity. When people say "look at this concurrent program I wrote" they rarely
[1] say "look at how well the code expresses the problem". What they
overwhelmingly say is "look at this benchmark".

[1] Joe Armstrong talks about how Erlang lets you represent processes more
like they happen in the real world. But that's a niche view. Most people think
about concurrency as a platform issue, where the platform is multicore
hardware or distributed systems, and otherwise wouldn't bother with it.

~~~
arohner
Clojure absolutely has threads, in the common meaning of the term. I use
futures all over the place in my code, and it's quite idiomatic.

(future (do (foo 1) (bar 2)) Runs the expression on another thread. Futures
intentionally compose well with the built-in clojure concurrency tools.

Clojure has no dislike of threads. It scorns locks, and code that is safe in
one thread, but unsafe in multi-threaded situations.

~~~
gruseom
Ah, thanks for the correction and teaching me something.

------
Mr_T_
I would love to see a video of the actual talk. I have searched for it
multiple times but I couldn't find anything.

~~~
enneff
We are still waiting on the production company that Heroku used to deliver the
goods, believe it or not.

------
seunosewa
I thought he was just giving an excuse for the abysmal multi-core scaling of
idiomatic Go programs.

~~~
KirinDave
Go scales quite well across multiple cores _iff_ you decompose the problem in
a way that's amenable to Go's strategy. Same with Erlang.

No one is making "excuses". It's important to understand these problems. Not
understanding concurrency, parallelism, their relationship, and Amdahl's Law
is what has Node.js in such trouble right now.

~~~
ryah
Trouble? Node.js has linear speedup over multiple cores for web servers. See
<http://nodejs.org/docs/v0.8.4/api/cluster.html> for more info.

~~~
KirinDave
You know, ryah; I like your work ethic, I like your enthusiasm, I think you're
a cool guy and it's great your project has so much traction.

But you say things like this and it worries me. Because a lot of people look
up to you and either you said this because you feel defensive about your
project or you said it because you genuinely don't understand the cases we're
talking about here. And this is a problem because a lot of people look up to
you and what you say, so when you say something as baffling as this response,
you run the risk of leading a lot of people astray.

I was sort of at a loss for how to reply in the time I have to spare for
Hacker News, but thankfully Aphyr did for me. But let me clarify what I said a
bit, since I was a bit terse: The problem Node.js has is a social one. A lot
of node hackers take the stance, "I thought Node.js solved the problems
threads presented,"
([https://groups.google.com/d/msg/nodejs/eVBOYiI_O_A/kv6iiDyy9...](https://groups.google.com/d/msg/nodejs/eVBOYiI_O_A/kv6iiDyy9ZwJ))
like there is a single axis of superiority and Node.js sits above the methods
that came before. But the reality is that Node.js is really just another
possible implementation in the Ruby/Python/Pike/-Perl-¹ space, and shares most
of the same characteristics as those languages.

So you have a lot of people who are aces at front-end programming in the
browser thinking they have a uniformly superior tool for tackling high-
performance server problems, but really they don't have that; they just have a
tool with familiar syntax. And so they fearlessly (and perhaps admirably)
charge into the breech of platform programming without realizing that the way
people scale big projects involves a lot of tools, a lot of thought about
failure modes, and a lot of well-established algorithms with very specific
tradeoffs.

And so this is Node.js's problem. It's just another gun in the gunfight, but
its community thinks its a cannon. In a world where high-performance parallel
VMs like Java or Erlang have very powerful and helpful languages like Clojure
or Scala on top, we're in a funny situation. It becomes increasingly difficult
to justify all these GIL-ridden implementations of languages.

Which is not to say these implementations don't have their place (and Node.js
is hardly the first javascript implementation outside of a browser), but
increasingly they are losing their place in the pieces of your code expected
to shuffle data around efficiently in the backend of modern distributed
applications.

¹ Correction, perl 6 doesn't plan share this behavior. What I read suggests
it's not done yet.

~~~
ryah
Node is popular because it allows normal people to do high concurrency
servers. It's not the fastest or leanest or even very well put together - but
it makes good trade offs in terms of cognitive overhead, simplicity of
implementation, and performance.

I have a lot of problems with Node myself - but the single event loop per
process is not one of them. I think that is a good programming model for app
developers. I love Go so much (SO MUCH), but I cannot get past the fact that
goroutines share memory or that it's statically typed. I love Erlang but I
cannot get the past the syntax. I do not like the JVM because it takes too
long to startup and has a bad history of XML files and IDE integration - which
give me a bad vibe. Maybe you don't care about Erlang's syntax or static
typing but this is probably because you're looking at it from the perspective
of an engineer trying to find a good way to implement your website today. This
is the source of our misunderstanding - I am not an app programmer arguing
what the best platform to use for my website--I'm a systems person attempting
to make programming better. Syntax and overall vibe are important to me. I
want programming computers to be like coloring with crayons and playing with
duplo blocks. If my job was keeping Twitter up, of course I'd using a robust
technology like the JVM.

Node's problem is that some of its users want to use it for everything? So
what? I have no interest in educating people to be well-rounded pragmatic
server engineers, that's Tim O'Reilly's job (or maybe it's your job?). I just
want to make computers suck less. Node has a large number of newbie
programmers. I'm proud of that; I want to make things that lots of people use.

The future of server programming does not have parallel access to shared
memory. I am not concerned about serialization overhead for message passing
between threads because I do not think it's the bottleneck for real programs.

~~~
rvirding
"I love Erlang but I cannot get the past the syntax."

I cannot understand this hangup about syntax. Syntax is the easiest thing to
learn with a new language, you just look it up. It's the semantics and usage
where the real problems are.

Erlang doesn't have static typing, it is dynamically typed. It has always been
dynamically typed.

And however you look at it doing highly concurrent systems with processes is
__much __easier. And who says that processes imply "parallel access to shared
memory"? Quite the opposite actually.

~~~
zem
syntax is the user interface of the language. it is one of the parts you have
to deal with on a constant basis, both writing and reading code. i can well
see how an unpleasant syntax can be a constant, grating annoyance when dealing
with a language. (i happen to really like erlang's syntax, but it definitely
has very different aesthetics from javascript's)

~~~
KirinDave
Honestly? Erlang's syntax is not _that_ bad. It's not great in that it has
prolog legacy with uncanny valleys to C-legacy left and right, but it's
effective.

Really the only really tricky thing for a newbie is strings.

~~~
zem
these things are very much a matter of taste. what does "effective" mean with
respect to syntax? for instance, lispers will argue that their syntax is
effective because of the things it lets you do, but that's orthogonal to
whether it's actually a pleasant syntax to program in, which comes down to
personal taste[1]. likewise, i love mlish syntax, but opa, a language steeped
in ml semantics, nonetheless switched to something more javascripty for their
main syntax because it proved more popular[2]. and read through the wrangling
over ruby versus python sometime - the two languages are very similar under
the hood, but their respective syntaxes are one of the things proponents of
each language complain about when they have to use the other.

[1] as larry wall famously said, "lisp has all the visual appeal of oatmeal
with fingernail clippings mixed in."

[2] <http://blog.opalang.org/2012/02/opa-090-new-syntax.html>

~~~
silentbicycle
It takes a lot of chutzpah for Larry Wall to say anything critical about Lisp
syntax. Larry Wall. _Come on_.

I'm not the author of the comment above, but I think Erlang's syntax is
effective in that it strongly emphasizes computation by pattern matching. If
you write very imperative code in it (as people tend to, coming from Ruby or
what have you), yes, it _will_ look gnarly. Good Erlang code looks
qualitatively different. There are pretty good examples of hairy, imperative
Erlang code being untangled in this blog post:
[http://gar1t.com/blog/2012/06/10/solving-embarrassingly-
obvi...](http://gar1t.com/blog/2012/06/10/solving-embarrassingly-obvious-
problems-in-erlang/)

The . , ; thing is a bit of a hack, admittedly -- I suspect that comes from
using Prolog's read function to do parsing for the original versions of Erlang
(which was a Prolog DSL), and reading every clause of a function definition at
the same time. Prolog ends every top-level clause with a period, not "; ; ; ;
.". (Not sure, but a strong hunch, supported by Erlang's history.) I got used
to it pretty quickly, though.

~~~
rvirding
As in Prolog ',' and ';' are separators: ',' behaves like an _and_ , first do
then and then do this (as in Prolog); while ';' is an _or_ , do this clause or
do this clause (again as in Prolog). '.' ends something, in this case a
function definition. Erlang's functions clauses are not the same as Prolog's
clauses which explains the difference.

It is very simple really, think of sentences in English and it all becomes
trivially simple.How many sentences _end_ in a ';'?. And you almost never need
to explicitly specify blocks.

Here <http://ferd.ca/on-erlang-s-syntax.html> are some alternate ways of
looking at it.

~~~
silentbicycle
Indeed. It makes sense to me. Are my intuitions about the origins of ;s
separating top-level clauses accurate?

(Hello, Robert! :) )

~~~
rvirding
Sort of. The syntax evolved at the same time we were moving from Prolog onto
our own implementation, which forced us to write our own parser and not rely
on the original Prolog one. The biggest syntax change came around 1991, since
then it has been mainly smaller additions and adjustments.

