
Go Concurrency Patterns: Pipelines and cancellation - enneff
http://blog.golang.org/pipelines
======
obilgic
The beauty of Go is, I have been writing these pipeline functions without even
being explicitly taught the pattern.

~~~
RobinUS2
Likewise, it simply makes sense if you have a proper understanding of channels
and go-routines.

------
Jabbles
Note the use of receive-only channels:

    
    
        func merge(cs ...<-chan int) <-chan int {
    

Inside the function you cannot send to a member of cs - it will give you a
compile-error.

Outside the function you cannot send to its return value.

An example to play with:
[http://play.golang.org/p/6AJLpZkLBe](http://play.golang.org/p/6AJLpZkLBe)

~~~
AYBABTME
I've cleanly used that in a lil' game I'm working on.

Players send their move to the game on a channel they expose through Move().
They receive regular updates of the game state on a channel they expose
through Update().

    
    
        type Player interface {
            Name() string
            Move() <-chan Move
            Update() chan<- State
        }

>
> [https://github.com/aybabtme/bomberman/blob/master/player/pla...](https://github.com/aybabtme/bomberman/blob/master/player/player.go#L30-L34)

The game reads the player moves, if moves are available[1]:

    
    
        for pState, player := range g.Players {
            if pState.Alive {
                select {
                case m := <-player.Move():
                    movePlayer(g, board, pState, m)
                default:
                }
            }
        }
    

>
> [https://github.com/aybabtme/bomberman/blob/master/bomberman....](https://github.com/aybabtme/bomberman/blob/master/bomberman.go#L250-L251)

The game sends each turn's update to players that are ready to listen for it:

    
    
        for pState, player := range game.Players {
            pState.Board = board.Clone()
            pState.Turn = game.Turn()
            select {
            case player.Update() <- *pState:
            default:
            }
        }
    

>
> [https://github.com/aybabtme/bomberman/blob/master/bomberman....](https://github.com/aybabtme/bomberman/blob/master/bomberman.go#L262-L269)

This leans to surprisingly clean implementations:

    
    
      * A websocket player [2]
      * A keyboard player [3]
      * A random AI player [4]
    

Note that the rest of the game (`bomberman.go`) is a hairy ball that needs
refactoring.

[1]: this, to prevent an unresponsive player from hanging the game, or an AI
player from unfairly computing longer than it's opponent

[2]:
[https://github.com/aybabtme/bomberweb/blob/master/player/pla...](https://github.com/aybabtme/bomberweb/blob/master/player/player.go)

[3]:
[https://github.com/aybabtme/bomberman/blob/master/player/inp...](https://github.com/aybabtme/bomberman/blob/master/player/input/input.go)

[4]:
[https://github.com/aybabtme/bomberman/blob/master/player/ai/...](https://github.com/aybabtme/bomberman/blob/master/player/ai/random.go)

------
songgao
> In Go, we can do this by closing a channel, because a receive operation on a
> closed channel can always proceed immediately, yielding the element type's
> zero value.
    
    
        for n := range c {
            select {
            case out <- n:
            case <-done:
            }
        }
    

This, is brilliant.

~~~
jbert
I wasn't quite getting that code when I first read it.

Now I can see it allows you do "discard" all the pending reads from the input
channel (and avoid writing to a possibly-closed 'out' channel), but wouldn't
it be better to break the loop in this case for an immediate exit?

i.e. go for 'interrupt' semantics, rather than 'drain'?

~~~
Sajmani
Yes, interrupt is better, and the article explains how to do that a few
paragraphs later:

    
    
            for n := range in {
                select {
                case out <- n * n:
                case <-done:
                    return
                }
            }
    

If you allow pipeline stages to interrupt their receive loop instead of
draining the inbound channel, then _all_ send operations need to be governed
by done. Otherwise the interrupted receive loop may block the upstream sender.

------
genwin
I use channels to single-file data to a SQLite db. Data gets pushed into
channels from multiple users / locations of the web app. A single goroutine
(concurrently running function) reads from the channels one channel at a time
and, for each channel, writes everything in the channel to the db in a single
transaction. SQLite isn't the best at concurrency. Go's channels help resolve
that issue nicely.

------
noname123
Cool. Can the concurrency guru's out there tell me what is the difference
between the "Go Concurrency Pattern" and the Java SingleThreadExecutor?
([http://docs.oracle.com/javase/6/docs/api/java/util/concurren...](http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/Executors.html#newSingleThreadExecutor%28%29))

    
    
      ExecutorService pipeline = ExecutorService.newSingleThreadExecutor();
    
      FutureTask<String> step1 =
           new FutureTask<String>(new Callable<String>() {
             public String call() {
               return "step1";
      }});
    
      executor.execute(step1); // execute single-step pipeline
      
      step1.cancel(true); // interrupt the task

~~~
songgao
I guess an obvious difference is that Go has these stuff built into runtime
and syntax, which makes those light-weight threads / tasks a first class
thing.

Furthermore, is there a way in "Java SingleThreadExecutor" and its related
components that allow automatic scheduling tasks on different threads? For
example, you are on a quad-core and you want to have 4 threads running
together, with millions of tasks mapped to 4 threads and dynamically balanced.

~~~
saryant
I believe that's what the ForkJoinPool is for.

[http://docs.oracle.com/javase/7/docs/api/java/util/concurren...](http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinPool.html)

------
alexchamberlain
I'm a C++ developer by day, and I haven't tried Go yet. This looks really
intresting. There is a massive gap for some Dataflow Programming based
languages; not sure Go is quite there yet, but we're moving in the right
direction.

~~~
AlexanderDhoore
The future of programming is somewhere in the field of dataflow and
(functional) reactive programming. It brings easy state to the declarative
programming world.

~~~
seanmcdirmid
Said often and probably false. Neither frp nor dataflow are very expressive.
Probably something still imperative but more reactive is the future.

------
aray
Great explanation in document form of these patterns. Also very succinct and
easy to reference.

For those that would rather read this in presentation form:
[http://talks.golang.org/2012/concurrency.slide#1](http://talks.golang.org/2012/concurrency.slide#1)

And for those that would rather watch a video presentation:
[https://www.youtube.com/watch?v=f6kdp27TYZs](https://www.youtube.com/watch?v=f6kdp27TYZs)

~~~
enneff
Both those links are in the original article, but they are not the same
content.

------
tete
Node.js has another beautiful way of doing this. It's called streams there.

[https://github.com/substack/stream-
handbook](https://github.com/substack/stream-handbook)

[http://nodejs.org/api/stream.html](http://nodejs.org/api/stream.html)

------
codygman
Interesting, I'll have to compare pipelines in Go to pipelines in Haskell.

~~~
signa11
> Interesting, I'll have to compare pipelines in Go to pipelines in Haskell

check out the "power series, power serious" paper by doug-macllroy. he does
that...

~~~
codygman
As jerf said, this is a little different since it's comparing to lazy io
rather than pipes or conduit.

