
Simplify Service Dependencies with Nodes - PaulHoule
https://blog.twitter.com/2016/simplify-service-dependencies-with-nodes
======
adamnemecek
This is like the fourth instance of computational graphs that I've seen
recently. Can someone recommend some resources on the underlying theoretical
ideas? It's not just graph theory, it's something like computational graph
theory or maybe data flow but neither seems to be at the level of formalism
I'd like.

~~~
PaulHoule
You can execute that kind of task with a RETE engine:
[https://en.wikipedia.org/wiki/Rete_algorithm](https://en.wikipedia.org/wiki/Rete_algorithm)

~~~
adamnemecek
Is there something more general?

~~~
cjslep
What do you mean by more general? Data in nodes in the RETE algorithm can be
anything.

I highly recommend reading "Production Matching for Large Learning Systems"[0]
as it was accessible enough for me to try to make a basic RETE implementation
while learning C++ and playing around with several other technologies I was
unfamiliar with[1].

[0][http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-
CS-95-11...](http://reports-archive.adm.cs.cmu.edu/anon/1995/CMU-
CS-95-113.pdf)

[1][https://github.com/cjslep/cpp-rete-
prototype](https://github.com/cjslep/cpp-rete-prototype) (awful, abandoned
prototype)

------
WhitneyLand
Maybe I'm missing something, how is this better than simple promises in JS?
This seem easy to reason about:

serviceA { Promise.All(b,c).then(...) }

serviceB { (d).then(...) }

serviceC { Promise.All(d,e).then(...) }

~~~
suls
Yes you are missing something. The purpose of this library is to decouple the
"description" from the "computation" itself. I usually call "separating the
_what_ from the _how_".

Essentially they construct an AST for whatever services that need to be
called. Most of the time they then simply "interpret" the AST with the lower-
level Finagle constructs. But, and that's actually the big point, they can use
_the same code_ with a different interpreter to generate a GraphViz
representation for visual inspection. This super helpful once your call graph
becomes more complex .. Another possibility is to write an optimizing
interpreter etc.

Anyhow, what I find a bit sad is that they didn't leverage the Scala part if
Finagle more .. this would be a prime example for a Free Monad ( _) based DSL.
That way they could have avoided writing 10s of classes with less than
fortunate names.

(_) might need a Free Applicative part as well since they inherently want to
have parallelism expressed in the language

~~~
wereHamster
Monad implies Applicative, so the "might need a free applicative part as well"
is already a given.

~~~
suls
Yes and no. With Free Applicatives you can express side-effects such as
parallelism .. not so with Free Monads afaik

------
jondot
So, between the lines - Twitter looks ahead to move to Java 8 and finally
ditch Scala?

~~~
Larrikin
Why ditch Scala completely?

~~~
jondot
Last I've heard, Twitter was "stuck" with their own version of Scala (was it
off 2.10?). Typesafe becoming Lightbend (i.e. not Scala focused anymore) for
doing business, Odersky moving all of his focus to Dotty, and core Scala
contributors leaving. That's my intuition of asking.

~~~
simono
Kotlin is a joke, but Scala's mismanagement and treatment of its contributors
might be its undoing. Source: #1/#2 top active community contributor (depends
on whether you count commits or locs in scala/scala) leaving Scala, because
I'm tired of their shit.

~~~
virtualwhys
Yeah, Scala's a bit of a mess right now, definitely not thrilled that the
flagship web framework (Play) won't be supporting Scala 2.12 until the 2.6
release (i.e. 4-6 months from now), which is absurd given that pretty much the
entire ecosystem has a 2.12 release at this point :\

As for typed alternatives, there's no substitute for Scala. Maybe Haskell or
OCaml if you leave the JVM, but otherwise it's pretty much a language
wasteland for anyone that's taken a dip in the ocean of Scala.

> because I'm tired of their shit

heh, think it goes both ways ;-) As an observer none of the powers that be
took a liking to you calling them out, justified or not (agree that community
contrib takes a back seat to Lightbend/EPFL).

Good luck wherever you've wound up (and thanks for the biased Either contrib
in 2.12, finally!)

~~~
simono
> As an observer none of the powers that be took a liking to you calling them
> out

The best way to avoid being called out for saying completely different things
in public and private is not saying different things in public and private.

If me politely asking for clarification about this sudden change of opinion is
"underhanded behavior", or "renouncing from future interactions" means "let's
find some flimsy pretext to contact your place of work to tell them what an
evil person you are" then Scala might have some issues attracting and
retaining contributors in the future.

~~~
virtualwhys
There's always Typelevel, sure you'd be welcome there as that's kind of the
anti-establishment wing of the Scala community ;-)

It's also thriving/taking off, tons of contributions from myriad brilliant
minds; for this reason I'm not too worried about Scala, it will continue on
with or without Lightbend steering the ship.

~~~
simono
Don't think so.

They won't be able to prevent the language from being run into the ground by
SIP/SLIP/SPP committees.

Anyway, getting told that I'm not qualified to tell people that their new
language extension ideas tick all the boxes of "bad ideas that we are
regretting and deprecating since years" after I have more or less managed
deprecations and removals for four major versions of Scala ... I guess they
need to find someone else to do my job now.

But given past experience, the people who are eager to add more and more
features are seldomly the people who clean up after themselves.

Most upcoming language proposals have extremely poor quality, ignore years of
lessons learned, repeat many mistakes of the past, reinvent things that have
been tried without success elsewhere and have no respect for design principles
that made Scala great.

So glad that my name won't be associated with this.

~~~
virtualwhys
Well, will be interesting to see how Dotty plays out.

DOT may have been proven sound, but compilers are complex beasts; implementing
DOT while preserving an upgrade path for Scala will likely be both difficult
and limiting (since Dotty will inevitably be shackled by Scala's past).

For now we've got Scala though, in a couple of years maybe Dotty.

~~~
simono
Dotty will inevitably implement the same features with the same fundamental
issues. This is not about some old stuff we all regret having in Scala, this
is about new low-quality proposals. When they get rubberstamped, they will end
up in both Scala and Dotty.

------
gigatexal
So I don't fully follow it all -- but I find it really fascinating. Perhaps
the real value at twitter is not the service twitter offers of letting you
communicate 140 characters at a time but the engineering that goes on behind
the scenes.

------
bozoUser
From the article..

> Nonetheless, Finagle still solved a huge problem for us as we now have real
> data dependencies and a more efficient execution engine. We gradually
> converted the Blender workflow code from batch style to Future style in
> 2013. It ran faster, the code became less error-prone but the readability
> was still not perfect. It was hard to follow, a pain to debug, tricky to
> test, and there were lots of duplicate function names.

from the git repo for Node...

> However, this library was written in Scala and isn't exactly Java friendly.
> It naturally involves a lot of callbacks and repeated function signatures.
> When it comes to waiting on multiple Futures, the code gets ugly very fast.
> Nodes is a Java library that aims to solve these problems, making the
> asynchronous code easier to read, to maintain and to test in Java.

what are they suggesting ? use Finagle for Scala and Nodes for java based
projects?

------
felixgallo
Seems like all the languages and frameworks are creeping slowly and cautiously
towards the Actor model. Come on in folks, the water's fine!

------
linkmotif
Stream processing, ladies and gentlemen! Kafka Streams, Apache Beam, and
everything else. Great to see so many people publishing their results for the
problems addressed here.

~~~
je42
there are not doing the stream processing are they ? this is still lots of
request / response.

~~~
je42
they are doing stream processing inside the same process.

------
oh_sigh
Please don't ask me why, but I'm going to go out on a limb and guess that one
of the people who "invented" this system at Twitter is a Xoogler.

~~~
eismcc
Actually I wrote the first version of this graph library 3yrs ago. It was to
help the search relevance team create reusable components, turn implicit
dependency management into a first class citizen and finally add a way to dump
the graph state at runtime or statically.

