
Tame.JS: Flow-control by the makers of OkCupid.com - petar
http://tamejs.org/
======
aston
I've been waiting for this sort of thing for the longest. The first time I saw
the callback passed to a callback of a callback's callback style in Node.js, I
wondered why the code couldn't look as nice as the async C++ I'd written at
OkCupid. From my lips to Max & Chris's minds... (telepathy!)

How long 'till the port to CoffeeScript syntax?

~~~
malgorithms
It's funny, when Max started programming Tame.Js a couple weeks ago, I
exchanged emails with a developer who tried to make something similar for
CoffeeScript who couldn't convince others it was needed. He said the common
response was that if you need something like Tame for async you're not "doing
it right." (Obviously we disagree.)

~~~
jashkenas
I'd be more than happy to explore the addition of Tame.js-style CPS to the
CoffeeScript compiler -- but there's a lot of prior work there already:

<https://github.com/jashkenas/coffee-script/issues/241>

<https://github.com/jashkenas/coffee-script/issues/287>

<https://github.com/jashkenas/coffee-script/issues/350>

 _Edit_ :

Things look a little less promising after running a simple test. This input
JavaScript:

    
    
        while (i--) {
          twait {
            fs.readFile("one");
            fs.readFile("two");
          }
        }
    

Gets compiled into this resulting "tamed" JavaScript:

    
    
        var tame = require('tamejs').runtime;
        var __tame_fn_0 = function (__tame_k) {
            var __tame_k_implicit =  {};
            var __tame_fn_1 = function (__tame_k) {
                if (i --) {
                    var __tame_fn_2 = function (__tame_k) {
                        var __tame_ev = new tame.Event (__tame_k);
                        var __tame_fn_3 = function (__tame_k) {
                            fs .readFile ( "one" ) ;
                            fs .readFile ( "two" ) ;
                            tame.callChain([__tame_k]);
                        };
                        __tame_fn_3(tame.end);
                        __tame_ev.trigger();
                    };
                    tame.callChain([__tame_fn_2, __tame_fn_1, __tame_k]);
                } else {
                    tame.callChain([__tame_k]);
                }
            };
            __tame_k_implicit.k_break = __tame_k;
            __tame_k_implicit.k_continue = function() { __tame_fn_1(__tame_k); };
            tame.callChain([__tame_fn_1, __tame_k]);
        };
        __tame_fn_0 (tame.end);
    

... not so nice to work with or debug. The general conclusion of that series
of tickets was that the code generation required to make this CPS
transformation work with all edge cases is a bit too hairy to be worth it on
balance. Depending on how much sequential async you're doing, YMMV.

~~~
maxtaco
It's on the list of todos to preserve the input line-numbering in the output
code. This would mean some changes to the code emitter, and also to the code
emitted to preserve the ordering of statements.

In tame/C++, debugging either compiler or runtime errors is straight-ahead, as
the developer doesn't need to examine the mangled code. Now if only JavaScript
had the equivalent of cpp's #line directive....

~~~
jashkenas
We're getting there ... maybe this summer.

<https://bugs.webkit.org/show_bug.cgi?id=63940>

<https://bugzilla.mozilla.org/show_bug.cgi?id=618650>

------
statictype
Ever since I first heard of CoffeeScript, I'd been hoping that features like
this would make it into the language. It's not realistic to wait for
javascript interpreters in the browser to catch up, but this would be a
perfect addition for a compiler like CoffeeScript.

------
pmjordan
The problem this solves is a serious one, in my experience, even though I find
their choice of syntax rather curious. Given that JavaScript 1.7 introduces
the _yield_ keyword, it would make sense to add support for that to V8 and
implement the infrastructure for concurrent asynchronous I/O around that as a
library. The concurrency aspect is, after all, orthogonal to the blocking vs.
callback situation, and can easily be done even when using callbacks, with a
single callback function called upon completion of all concurrent I/O. I
believe the Dojo framework provides such a utility, and I wrote my own
simplistic stand-alone mini-library for exactly this a while back. [0]

I've run into the problem of endless chained callbacks in C, where it's much
worse due to the lack of nested functions, let alone closures or garbage
collection.[1] I ended up using the switch block "coroutine" hack [2] for the
worst cases, along with dynamically allocated "context" structs to hold
"local" variables. A proper macro system would have helped transform blocking
code into CPS form. I tried to integrate a SC [3] pass into our build, which
could have done it, but ran into all sorts of practical/yak shaving problems,
so I ended up with the C preprocessor macro/switch solution for now. In user
space, explicit stack-switching with something like _swapcontext()_ is
probably preferable, if you can get away with it, but in the kernel this is
rather problematic.

[0] <https://github.com/pmj/MultiAsync-js>

The reason I wrote my own was because I originally needed it in Rhino, the
JVM-based JS implementation, and I couldn't find anything similar that worked
there.

[1] Yes, there are garbage collectors that work with C, but to my knowledge,
none of them can be used in kernel modules. In any case, the other 2 issues
are worse and aren't solveable within the language via libraries.

[2] <http://www.linuxhowtos.org/C_C++/coroutines.htm>

[3]
[http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.htm...](http://super.para.media.kyoto-u.ac.jp/~tasuku/sc/index-e.html)

~~~
maxtaco
My first thought for implementing tame.js was with yield, but V8 doesn't
currently support it (though it's reserved as a "future" keyword). A direct
use of yield (without anything like twait) would probably make node code look
more like Python/Twisted code, which while better than the status quo, still
can get unmanageable in my experience.

Agreed that twait conflates concurrency with blocking/callbacks, but it my
experience, it's a natural and useful combination.

~~~
bdarnell
I think you could do something manageable with yield alone (at least with
python-style generators). I've been meaning to try something like this with
tornado. The general idea is that yielding would either immediately produce a
callback or asynchronously "wait" for a previously-generated callback to be
run. It would look something like this:

    
    
        doOneThing(yield Callback("key1"))
        andAnother(yield Callback("key2"))
        res1 = yield Wait("key1")
        res2 = yield Wait("key2")

------
snprbob86
Looks a lot like C# 5's await/async keywords:

<http://blogs.msdn.com/b/ericlippert/archive/tags/async/>

Cool to see growing interest for this at the language level.

~~~
contextfree
It looks even more like F# async workflows, which compared to the C# async
feature have the advantage of being implemented in the current shipping
version rather than the next one.

------
reustle
I've been very happy with "parallel" in the async library by caolan
<https://github.com/caolan/async>

------
sjs
Why hasn't anyone brought up error handling yet? What happens when an error is
thrown inside a twait block? What happens when 2 errors are thrown inside a
twait block?

Tame.js looks nice in that it's very simple to learn but ~300 lines of plain
old JavaScript[1] can give you a general purpose deferred/promise library with
more flexibility should you need to do something other than wait on N async
operations and then use the results.

[1] [https://github.com/heavylifters/deferred-
js/blob/master/lib/...](https://github.com/heavylifters/deferred-
js/blob/master/lib/deferred.js)

------
geuis
In what significant ways is this different from deferreds/promises/futures?

~~~
voidfiles
I was wondering the same thing. If the spec evolves tools to handle this
stuff, thats one thing, and of course this could be an example solution, but
right now I think the problem has a solution without adding extra keywords to
the language.

~~~
bialecki
I would've thought the same thing until I wrote substantial async code in JS
and Python.

It's not that it doesn't work and you can't do it, it's that the code becomes
a mess and their example gets at that. It probably doesn't seem like a big
deal, but when you constantly have to pass around callbacks and chain requests
together you get this feeling the code could look a lot cleaner than it is.
You want asynchronous behavior but with synchronous syntax.

This isn't possible without adding _something_ , and, having seen a lot of the
solutions out there, it's nice to see someone take a stab at it by changing
the language. The cost is huge (preprocessing, etc.), but, speaking from
experience, the simplicity of the code you write might make the change worth
it. You get the feeling in 5-10 years this will be a solved problem, but I'm
not sure any of the solutions out there yet will be the accepted solution.

------
Pewpewarrows
This seems neat, but after reading through it twice I can't seem to understand
how this provides any advantage over just using Deferreds. Someone care to
enlighten me?

------
va1en0k
That reminds me of monads in Haskell (they can solve this problem (and other
as well) matemagically). I've seen somewhere a proposal to add them into
Javascript, but I doubt the idea will be loved by public.

(By "add them to JS" I mean some syntactic sugar, not a library)

~~~
pmjordan
Proposal? JavaScript 1.7 introduced generators circa 2006, which I do believe
lets you implement something like this (in addition to a bunch of other cool
things).

[https://developer.mozilla.org/en/New_in_JavaScript_1.7#Gener...](https://developer.mozilla.org/en/New_in_JavaScript_1.7#Generators)

Unfortunately, V8 doesn't support generators, as far as I can tell.

~~~
va1en0k
sorry? generators? well, I like generators, but what's your point?

~~~
pmjordan

      yield;
    

Will block execution until the generator is resumed and return control flow to
the caller. I haven't actually tried it, but it should be pretty
straightforward to wrap a generator function in such a way that it's driven
(resumed) by callbacks from async I/O calls.

------
janetjackson
This is the wrong solution to the problem, and it's implemented poorly.

Use a proper control-flow ( not "flow-control" ) library like
<https://github.com/caolan/async>.

Furthermore, why would you write an entire custom js parser for this? Why not
use some of the many many pre-existing ones that are much more stable, more
developed, and well supported.

~~~
__david__
Seriously? The library solution is pretty ugly--though it's a good solution if
you are absolutely dead set against compiling your "javscript". Tame, being a
code transformer, makes the equivalent code _so_ much more readable and clean
looking.

How on earth is readable, clean looking code "the wrong solution to the
problem"? I would argue that it's almost always the _right_ solution to a
problem.

------
frankdenbow
Slightly Unrelated: There was a site recently on hn that was a listing of
various js libraries, like this one, on one page. What was it?

~~~
matthiaswh
There are two that fit your description and are really useful:

<http://www.everyjs.com/>

<http://microjs.com/>

~~~
frankdenbow
thanks! microjs was what i was thinking of

------
yaix
This is nice on the browser, but not very useful in nodeJS.

twait{} will block and stop my nodeJS process from doing anything else.

It would be more useful if I could give twait{} a callback to fire when all
its async events completed. Then my nodeJS process could do other stuff while
waiting for a twait{} bundle to finish.

~~~
malgorithms
No, that's not the case. twait won't block your Node process from handling
other events.

~~~
yaix
Isn't it waiting to execute anything that follows after a twait code block?

If so, then it is blocking. Otherwise, how do you manage to let other code be
executed, except the code you don't want to be executed until all functions in
the twait block have returned?

The would need to be a callback attached to the twait block, but there isn't.
So it's blocking.

Because that is it's purpose, to block further executing until all data from
the non-blocking functions have returned.

~~~
baudehlo
From looking at it, each twait block is blocking as a unit, but other code
elsewhere from the twait block will still run.

------
teyc
Also see JSCEX, which uses the C# keyword await

[http://groups.google.com/group/nodejs/browse_thread/thread/3...](http://groups.google.com/group/nodejs/browse_thread/thread/337f83c028f371c2)

------
lzm
The next version of Visual Studio will have something similar:
<http://msdn.microsoft.com/en-us/vstudio/async.aspx>

------
trungonnews
So the code we write is clean and easy to understand, but the debugger only
work with the compiled version of the code?

In another word, write in C, and debug in assembly...

------
starwed
There's a bug in huntMen. _if (! is_vamp)_ should instead be _if (
is_vamp)_... :P

------
tjholowaychuk
why not just use node-fibers?

~~~
maxtaco
A key advantage of fibers of course is that they preserve exception semantics,
whereas tame can't in all cases. I'm not too interested in reviving the
stalemated thread v. event religious war. I prefer explicitly-managed events,
but if others prefer a more thread-like API, by all means....

~~~
jrockway
In the end, both are the same thing; managing action-specific state in an
application-defined per-action data structure. With OS threads, you let the OS
manage the state instead. This can be inefficient with a large number of
actions, because most of the state kept has nothing to do with the application
itself; it's OS bookkeeping overhead.

------
diamondhead
For those who looking for the JavaScript way of the examples;
<https://gist.github.com/1090228>

