
An introduction to timely dataflow, part 2 - mrry
https://github.com/frankmcsherry/blog/blob/master/posts/2015-09-18.md
======
amelius
What advantage does a dataflow language have over a functional language? And
how is laziness handled (i.e., does it happen that in a dataflow language,
tokens are being sent unnecessarily)? Is it easy to perform memoization in a
dataflow language (to avoid performing the same computation twice)?

~~~
frankmcsherry
I don't want to speak for all of dataflow-dom, but the main differences that I
see (and exploit) are that data-parallel dataflow languages isolate control
flow into small independent regions, making the larger computation data-
driven. This does mean things are eager rather than lazy, but it also makes
things much easier to parallelize (because of the independence) and much
easier to incrementalize.[0]

[0]: [https://github.com/frankmcsherry/differential-
dataflow](https://github.com/frankmcsherry/differential-dataflow)

------
erickt
I love these articles, thanks! I can't wait for rust to get more involved in
this space. How do you plan on doing error handling in timely? Do you expect
it to be more along the lines of MPI, where you restart from the last
snapshot, or MapReduce/Spark, where lost work is transparently recomputed?

~~~
frankmcsherry
It's a good question. I'm punting for the moment because it isn't clear that
there is one true solution that makes everyone happy. So, let's say MPI then.
:)

I think the more optimistic version is probably that the signals Timely
Dataflow (the model) gives should make tasteful fault-tolerance logic easier
to write. Michael Isard did some work here just before leaving MSR.
Realistically, errors haven't been a big problem yet (thanks Rust!) and it's a
research prototype for understanding efficient data-parallel compute, so not
too bothered yet.

The MR/Spark approach comes with lots of hidden costs in the programming
model; I'm not clear on when these costs make sense. Using lots more resources
so that you survive being killed because you were using too many resources
seems like a not-best solution.

Some of the layers on top have better fault-tolerance properties. For example,
differential dataflow operators are functional and have append-only logs as
their internal state; I could see that stack ending up with more transparent
fault-tolerance properties, though not yet.

------
amirouche
It seems like timely dataflow relates to Tinkerpop Gremlin graph traversal
language. You build a pipeline a "steps" and those get executed over the input
data one after the other.

How timely dataflow can be compared to Gremin DSL?

