
Rich Hickey – Inside Transducers [video] - tsm
https://www.youtube.com/watch?v=4KqUvG8HPYo
======
Blackthorn
Even if you forget entirely about Clojure for a second, Rich has this very
rare gift to take a complicated subject and make it easy for the audience to
understand.

~~~
lomnakkus
This is a double-edged sword. Forgive me a small anecdote: I once had a
wonderful tutor who could explain any advanced concept in highly intuitive
terms and it just made sense... until I got out of the classroom. At which
point I'd just have this feeling of "Hang on, whaaaaaa...?". The first time I
attempted the exam in this particular subject matter, I failed miserably
(rightly so). Having learned my lesson, I went back to study the actual
_source material_ more thoroughly instead of mostly just listening to the
probably-best-educator on the subject. That time I _actually_ understood the
material and passed with flying colours. (I still think the particular
educator had a major role in me passing at all, but I digress.)

That that for what you will, but please be aware that a "gift for
simplification" sometimes is a double-edged sword and can leave the audience
_thinking_ that they've understood when, actually, they haven't.

I'm not saying that's the case here, just something to be wary of.

~~~
agumonkey
I had similar experiences many times, and I don't know what's best, looking at
an abstraction for weeks until you get the "ohh they just meant this" or
knowing in advance it's not that obscure and then work to imprint said
abstraction deeper.

~~~
lomnakkus
Ever since that experience, I've tried to _somehow_ try to practice working
with "X" in some concrete way, even though it might feel like a classic "math
problem" situation. Personally, I find it helps me get a grasp on "X",
whatever it might be. (Mind you, this is just personal experience so YMMV.)

------
ultimape
For those not familiar with Clojure, here's a great demonstration of the
concept done up in JavaScript: [http://jlongster.com/Transducers.js--
A-JavaScript-Library-fo...](http://jlongster.com/Transducers.js--A-JavaScript-
Library-for-Transformation-of-Data)

~~~
jonahx
From that link:

"The reduce function is the base transformation; any other transformation can
be expressed in terms of it (map, filter, etc)."

This seems so obvious in retrospect -- I can't believe I had never made that
connection before.

~~~
ultimape
I had a similar epiphany while learning about regular expressions in Perl.
That connection with text processing is what helped me understand list
comprehensions. They seem strikingly similar.

For JS, I personally prefer lo-dash for this kind of work, or dropping in
polyfills from MDN.

For some context, here are .map functions applied to normal arrays
[http://jsperf.com/native-vs-array-js-vs-
underscore/54](http://jsperf.com/native-vs-array-js-vs-underscore/54)

I've been looking for similar libraries that work on typed-arrays because they
are so much more efficient when working with web-workers or with raw canvas
data. My attempts at hacking it in feel like they are just bad ideas:
[http://jsperf.com/float32array-map/2](http://jsperf.com/float32array-map/2)

~~~
grayrest
Most people use underscore/lodash for this stuff. The difference is that the
js transducers libraries don't create intermediate arrays, only do enough work
to produce the requested output, and work on top of anything that can be
coerced into the iteration protocol. I've seen demos of using Facebook's
Immutable JS, CSP.js [1], and I don't see why you couldn't put them on top of
Typed Arrays or a FRP library like Kefir.

[1] [http://jlongster.com/s/nationjs-slides/](http://jlongster.com/s/nationjs-
slides/)

~~~
ultimape
Yeah, the overhead of creating intermediate typed arrays is exactly what I'm
trying to avoid with this.

~~~
jdd
Lo-Dash 3.0 will have support for lazy evaluation in its chaining syntax that
supports shortcut fusion and avoids intermediate arrays as well.

~~~
ultimape
Do you know if they work with typed arrays?

------
ultimape
Here's another video by Rick Hickey about transducers:
[https://www.youtube.com/watch?v=6mTbuzafcII](https://www.youtube.com/watch?v=6mTbuzafcII)

\- it was referenced in this great post detailing the transducer notion with
Clojure, Scala, and Haskell:
[http://blog.podsnap.com/ducers2.html](http://blog.podsnap.com/ducers2.html)

------
charlysisto
So Transducers generalize the usage of Enumerable functions such as map,
reduce, filter, flatMap (...).

Please could someone tell me if this is conceptually different from ruby's
Enumerable module which only needs the class its included in to implement
`each` so anything can be enumerable ? Or is it a similar but just in
translated to the FP world ?

~~~
YuriNiyazov
Yes; There are differences.

1\. Transducers are parallel under the covers. Since the expectations is that
the code that the predicate or mapping functions that you pass into map,
filter and reduce are pure (no variables are changed, no state is modified,
just a calculation that is only dependent on arguments) the parallelism is
hidden away from you, but it's there. Ruby's Enumerable can't do parallelism

2\. When you compose transducers, there are no intermediate sequences
generated. The simplest example is this:

In Ruby: [1, 2, 3].map {|x| x + 1}.map {|x| x+ 2} will generate an
intermediate array [2, 3, 4] after the first map, and then will generate [4,
5, 6] when it has evaluated the whole expression.

In Clojure (I am sure I got the syntax wrong for this one) (transduce (map #(+
2 %) (map inc)) [1 2 3]) will create an intermediate mapping function that
will first increment, then add 2 to its argument, and then will map once using
that intermediate function.

~~~
cfeduke
Transducers _can be_ parallelized (TODO) but some transducer implementations
do contain state, like `take` and therefore cannot be parallelized.

The big thing is no intermediate results.

~~~
YuriNiyazov
If you watch the first Rich Hickey Transducers talk, he shows that some
transducers (the ones that rely in the underlying reduce implementation that
uses fork/join and assumes that reducing the collection is associative) are
already parallelized. I agree with you regarding `take`

~~~
cfeduke
IIRC from yesterday's talk he was explaining that these will be easy to
parallelize, and its slated for 1.7. I had thought they were already parallel
myself until I saw the talk; and possibly some of them are, but the item is
still Open.

[http://dev.clojure.org/jira/browse/CLJ-1553](http://dev.clojure.org/jira/browse/CLJ-1553)

~~~
YuriNiyazov
I was just talking about the "reduce fold" being already parallel.

------
nickik
Again a more technical talk. I really liked it. Specially the end bit about
pipelines, that seams like it would be really usful.

Transducers overall are quite intressting, I am exited to see what other
transducer context people come up with.

~~~
moomin
It's not that clear there will be one. Transducers have three steps: begin,
during and end. In practice, start is only used for reduce operations and
outside of reduce operations end can only be used for its side effects.

There's also definitional issues e.g. can you have an async transducer?

~~~
cfeduke
Yes you can have async and blocking transducers. Forward to the point in the
video about channels for a discussion.

~~~
moomin
But if you use an async transducer on a channel, it fails. (Actual tested
code, sadly.)

Which might be okay, but my basic point that it's definitionally fuzzy still
holds.

------
sitkack
These aren't materially different from itertools in Python.

------
kushti
For those who know Russian, here's awesome critics of "transreducers"
[http://thedeemon.livejournal.com/87320.html](http://thedeemon.livejournal.com/87320.html)

~~~
vanderZwan
Could you try to give a summary for those who don't?

~~~
juskrey
The summary: "Hickey did not invent FP, how dares he?"

~~~
nickik
People keep attacking him whenever he interduces something. Why? Every time he
releases something, he also shows from what research it originated. Go back to
the first transducer talk and you will see the references.

~~~
lomnakkus
Personally, I don't mind him too much even though I'm a typeful-programming
weenie. That said, he does have a tendency to somewhat unfoundedly say "...
and you couldn't do this in a typed language" (or at least allude to it)...
whereas people again and again show that, yes, it could be done in a
statically typed language. In fact, this is trivially true in a sense as shown
by Bob Harper[1] another person who is undoubtedly hugely influential, but who
I have trouble with because of his obvious bias and trollish ways.

(I think Rich is absolutely _spot on_ on many of the more abstract things
about or industry/craft, but this is just one of those details that irks me.)

[1] [http://existentialtype.wordpress.com/2011/03/19/dynamic-
lang...](http://existentialtype.wordpress.com/2011/03/19/dynamic-languages-
are-static-languages/)

~~~
ajanuary
In the strangeloop talk his argument seemed to be that you couldn't use types
to describe the entire and exact contract of transducers. Which is true (and
true of almost anything), but you can at least describe a significant enough
chunk of it to reduce errors.

As always it's a trade off. As you try to capture more of the contract, you
massively increase complexity, sometimes to the point of reducing usability
(i.e. staring at an obtuse type error, or trying to work out how to do basic
things with a function just from a Scala type)

It's my interpretation that "you can't type transducers" is an invitation to
think more deeply about types and the limitations in what they can express,
mixed in with some "get off my back about creating a dynamically typed
language".

~~~
seanmcdirmid
Isn't the problem with transducer typing is that transducers aren't very
functional? In one of your posts you wrote:

    
    
       System.in.setTransducer(lowerAlpha);
    

In the LINQ/functional origami world, you could only produce new streams, not
transform an existing stream in place. Because a new stream is computed, a new
type can be computed. However, from what I've seen with transducers, they
would be limited to basically Func<T, Predicate<T>, T> signatures.

~~~
lomnakkus
That's part of it, but of course you can embed monadic types/effects within
the appropriate combinators. (I think "tel" showed an example of this in the
original "transducers" post)

