
Composable Pipelines [in Go] Improved - samuell
https://blog.gopheracademy.com/advent-2015/composable-pipelines-improvements/
======
seliopou
For those that are interested in libraries like these, OCaml has an
implementation that written by Jane Street called Pipes[0]. It's been around
for years, helps trade billions of dollars a day, is written in a language
that actually has parametric polymorphism, and a configurable notion of
"pushback". I've written about it before[1].

[0]: [https://ocaml.janestreet.com/ocaml-
core/111.28.00/doc/async_...](https://ocaml.janestreet.com/ocaml-
core/111.28.00/doc/async_kernel/#Pipe)

[1]:
[https://news.ycombinator.com/threads?id=seliopou&next=980930...](https://news.ycombinator.com/threads?id=seliopou&next=9809300)

~~~
PopsiclePete
A lot of cool concepts come from languages like OCaml and Haskell. But for
better or worse, we live in a world where simple languages rule the day (C++
being one notable exception), and if we get lucky, we get _some_ of cool
OCaml/Forth/Haskell/LISP/Erlang features to play with, in our simple little
worlds with our simple little toy languages like Java and C# and Go and
Javascript... it's sad in a way, but it's reality. I'd much rather write F#
(basically .NET-hosted OCaml) than C#, but no luck finding gainful employment
with that...

------
hb42
Without any parametric polymorphism, this kind of utility functions are hardly
composable at all.

At best these are copy-pastable patterns.

~~~
samuell
This sounds like an interesting observation, but could you please elaborate a
bit more?

~~~
catnaroek
With parametric polymorphism and generalized algebraic data types, you could:

(0) Give every process in the pipeline could its own input and output type,
rather than forcing all processes to use strings.

(1) Enforce, statically, that only "compatible" processes can be composed. For
instance, if `Foo` is a process that outputs integers, and `Bar` is another
process that takes strings as input, then `Foo` can't feed its output to
`Bar`.

However, this is involves some rather tricky type hackery (so-called "type-
aligned sequences"). Hardly a good fit for a language with Go's design goals.

~~~
samuell
Ok, I think I see what you mean.

Well, I think unfortunately my sketchy code examples maybe did not cover the
idea in enough detail.

What I'm doing in practice is to create different struct types for each type
of process I use, which all are of a "base interface" called "process" (mainly
for the Run() method, so that the pipeline component can call it).

An example of this in action is shown in this code example on the Go
playground:
[http://play.golang.org/p/voUfPGQulf](http://play.golang.org/p/voUfPGQulf)

The typing of channels for each process struct type should now enforce what
components can be connected together (you will not be able to assign a "string
chan" to a "[]byte chan" field, for example).

Did this answer the question, or do you see further problems here?

~~~
catnaroek
Ah, okay. I misunderstood the intention of your original example, then. My
bad.

In order to prevent further embarrassing myself, I'm going to tell you how I
(currently) understand your design, and, if I'm wrong, please tell me: Your
"pipeline" isn't actually connecting the processes to each other. You _first_
connect the processes to each other, and _only then_ add them to a "pipeline",
whose primary (only?) responsibility is to ensure that each process runs in a
different goroutine.

This much doesn't really need parametric polymorphism, let alone generalized
algebraic data types. However, calling it "pipeline" is kind of misleading,
because the actual "piping" (connecting processes to each other) is handled
elsewhere. In fact, using your "pipeline" abstraction, you could perfectly
well run a bunch of processes that aren't connected to each other at all.

Is my understanding correct?

~~~
samuell
> Your "pipeline" isn't actually connecting the processes to each other. You
> first connect the processes to each other, and only then add them to a
> "pipeline", whose primary (only?) responsibility is to ensure that each
> process runs in a different goroutine.

Exactly!

> This much doesn't really need parametric polymorphism, let alone generalized
> algebraic data types. However, calling it "pipeline" is kind of misleading,
> because the actual "piping"

Ah, yes, didn't think about that, but yeah, but that's a good point! I should
consider a better name for this ... Thanks for pointing it out!

------
secure
The example code doesn’t actually compile: [https://play.golang.org/p/Fv-
Sf1OJdH](https://play.golang.org/p/Fv-Sf1OJdH)

I think it should read: [https://play.golang.org/p/vrwEkHy-
DR](https://play.golang.org/p/vrwEkHy-DR)

~~~
samuell
Thanks! I have pushed a lot of fixes now and sent a PR.

In the meanwhile, please see this working example on the Go playground:
[http://play.golang.org/p/voUfPGQulf](http://play.golang.org/p/voUfPGQulf)

------
ceocoder
Hey 'samuell, I think you meant,

    
    
        func NewProcess() *A-Process {
         ....

~~~
samuell
Sorry about all of this, I will push fixed code examples!

~~~
samuell
I have pushed a lot of fixes now and sent a PR.

In the meanwhile, please see this working example on the Go playground:
[http://play.golang.org/p/voUfPGQulf](http://play.golang.org/p/voUfPGQulf)

------
hden
Just use a transducer.
[https://www.youtube.com/watch?v=6mTbuzafcII](https://www.youtube.com/watch?v=6mTbuzafcII)

~~~
samuell
Will have a look!

------
whateveracct
scalaz-stream does all this and way more and way better while actually being
"composable"

