

Fixing the callback spaghetti in node.js - koush
https://github.com/koush/node/wiki/%22async%22-support-in-node.js
node.js is a fantastic platform, but it suffers from your code quickly turning into callback spaghetti and a mess of functions.
I modified the V8 parser to support the await keyword that allows C# to handle this quite elegantly.
======
SomeCallMeTim
Very cool, but it's too bad you have to deal with all that to begin with.

I've been using LuaJIT embedded in Nginx (LuaNginxModule). Lua supports
coroutines, so a function can just yield. Here's a brief example:

    
    
        -- Query a database using an http backend.
        -- Yields and handles other requests until the reply is complete.
        local record = util.getUserRecord(userId)
    
        -- Send some text to the client. Yields control while
        -- the actual transfer is in progress.
        ngx.print( "Result=" )
        
        -- Send the result, encoded as JSON, to the client
        -- Again, this call doesn't block the server.
        ngx.print( cjson.encode(result) )
        
    

With code like the above I can easily handle into the thousands of concurrent
connections per second on the lowest end Linode VPS node available, with
barely any load on the box -- and I'm told it should be able to handle 40k+
connections per second, if I were to do any tuning. Oh, and I have only 512Mb
of RAM, which it doesn't even get close to under load. And the longest request
took less than 500ms at high load.

I've been using OpenResty [1] which has the Lua module and a bunch of others
all configured together. Works great, and I can't complain about the
performance.

Someday I'm sure I'll hand the maintenance of this off, and then I might
regret not using one of the "popular" frameworks. But the code is SO
straightforward using this stack -- and what I'm using it for is so simple --
I think not.

[1] <http://openresty.org/>

~~~
corysama
IIRC, the original plan for Node involved coroutines (maybe even Lua?). But,
they were scrapped because they introduced a lot of the same problems as full
threading. You never know if the state of the world is going to change in ways
unrelated to the function you are about to call because some deep-nested
subroutine might yield.

~~~
eis
The real reason will be that V8 does not support co-routines.

~~~
yesbabyyes
I think corysama is referring to Ryan Dahl experimenting with Lua (as well as
C and Haskell) before settling on JS/V8.

 _I had several failed private projects doing the same on C, Lua, and Haskell.
Haskell is pretty ideal but I’m not smart enough to hack the GHC. Lua is less
ideal but a lovely language - I did not like that it already had a lot of
libraries written with blocking code. No matter what I did, someone was going
to load an existing blocking Lua library in and ruin it. C has similar
problems as Lua and is not as widely accessible. There is room for a Node.js-
like libc at some point – I would like to work on that some day.

V8 came out around the same time I was working on these things and I had a
rather sudden epiphany that JavaScript was actually the perfect language for
what I wanted: single threaded, without preconceived notions of “server-side”
I/O, without existing libraries._

[http://bostinnovation.com/2011/01/31/node-js-
interview-4-que...](http://bostinnovation.com/2011/01/31/node-js-
interview-4-questions-with-creator-ryan-dahl/)

~~~
viscanti
The threading model in Haskell (GHC) would be ideal, as it's essentially
abstracted away from the developer. There's certainly the potential for a
performance advantage there as well as easier to maintain code. The advantage
Node.js has, is that the vast majority of web developers already know
javascript. It doesn't need to be the "perfect" server-side implementation, it
just needs to be good enough. It's success is due to it's easy of
accessibility, something he would have missed out on had he gone the Haskell
path.

~~~
gregwebs
Its not just potential - all the major Haskell web frameworks can handle more
concurrent requests than node.js and scale to multi-core.
[http://www.yesodweb.com/blog/2011/03/preliminary-warp-
cross-...](http://www.yesodweb.com/blog/2011/03/preliminary-warp-cross-
language-benchmarks)

I don't think node.js is good enough because you have to deal with the issue
being discussed here. In Haskell, as you said, you just write normal code with
no worries about mutable state messing with your thread.

------
jorangreef
I have been writing Node since late 2009 and Javascript with Rhino before that
and Javascript with Rails before that. Before Node, I was used to linear
program execution and was hesitant to switch to writing asynchronous code.

My mental model of code was one-dimensional: do this, then do that, then do
that. So I was used to exception handling for errors, and my programs were
like trains on a track. This was a comfortable abstraction, but the way in
which my programs were written did not reflect what they were doing in the
real-world: reading from disk, reading from the network, waiting for
something, doing something, responding to something, receiving something in
chunks.

Now, looking back, I prefer writing code that reflects what my programs are
doing. There is more headspace, another dimension, no longer one thing after
another, but a stack of things hovering and happening at the same time,
interacting with each other, moving forwards through time.

Now, instead of trying to make concurrent work appear non-concurrent, I prefer
to embrace concurrency and see how I can write for concurrency. This is almost
certainly different to the way in which I would have written synchronous code.
Code for me is less complex now and shorter now than when it was synchronous.
It feels richer and more descriptive of the work being performed. It's also
faster and more reliable. In essence, I have learned to write better
concurrent code.

~~~
saurik
FWIW, "Javascript with Rhino" supported co-routines, and frameworks like
Apache Cocoon took advantage of this to play tricks similar to Smalltalk's
Seaside. (That said, you might have meant something different, as Node and
Rails are both web tiers, whereas Rhino is an engine like V8.)

------
glenjamin
Callback spaghetti is a sign that you're doing something wrong.

The first example from this page shows a request handler initialising a
database connection and then executing a query. That's terrible separation!

Callbacks "spaghetti" actually does a great job of highlighting when you're
not abstracting enough, any more than about 5 indents and you should be
seriously considering refactoring your approach.

Thus

    
    
        app.get('/price', function(req, res) {
          db.openConnection('host', 12345, function(err, conn) {
            conn.query('select * from products where id=?', [req.param('product')], function(err, results) {
              conn.close();
              res.send(results[0]);
            });
          });
        });
        

becomes something like

    
    
        app.get('/price', function(req, res) {
          products.fetchOne(function(err, product) {
            res.render('product', { product: product });
          });
        });
    

Also, as other posts on this page mentioned, try{}catch{} is not how errors
are handled in Node.JS, plenty of async operations will gain a fresh stack,
and cannot be caught in this way.

~~~
moe
_becomes something like_

You comfortably omitted the error handling (admittedly the original snippet
didn't have that either).

However in production code it's not optional, and that's where node.js code
tends to get really nasty.

~~~
glenjamin
On this point I don't disagree, for this case you'd expect the data access
layer to deal with all the database errors and return something useful to the
request layer:

    
    
        app.get('/price', function(req, res) {
          products.fetchOne(function(err, product) {
            if (err) {
              res.render('error', { error: err });
            } else {
              res.render('product', { product: product });
            }
          });
        });

~~~
moe
_you'd expect the data access layer to deal with all the database errors and
return something useful_

There you nailed the problem. It's a constant headache, especially since
library authors have different ideas about the format and semantics of
"something useful".

Before you know it you have re-invented your own half-baked exception-
framework, to normalize/wrap/re-throw those ErrBacks. And then a week later
you notice that rollback/retry and event cascades need a whole new level of
treatment.

There's a reason why mature event-frameworks such as Twisted have never quite
taken over the mainstream. It's sad to see node burn all its powder on a niche
programming-style that just doesn't fly for the majority of applications.

------
tlrobinson
I really like this idea, but it doesn't seem to do error handling correctly.

This is not going to catch most errors occurring in an asynchronous APIs:

    
    
        try {
          asyncOperation(function(err, result) {
             // ...
          });
        } catch (e) {
          // ...
        }
    

Most errors will occur asynchronously, thus the convention of "err" being the
first argument to asynchronous API callbacks in Node.

Unless this supports that convention, your code should actually look like:

    
    
        async function magic() {
          try {
            // code here
            await err, bar = doSomething();
            if (err) {
              throw err;
            }
            // more code here
            await err, boo = doAnotherThing();
            if (err) {
              throw err;
            }
            // do even more stuff here
          }
          catch (e) {
            // handle the error
          }
        }
    

...or something similar. It's better than the alternative, but not great.

This certainly could support the "err" first convention, but APIs that don't
use that convention wouldn't work correctly.

~~~
tilgovi
Not necessarily. A callback function with the standard signature under the
hood is implied. If the author is controlling flow into and out of these
execution contexts there is no reason an error passed to the implicit callback
cannot cause an exception to be thrown into the original execution. Throwing
exceptions into coroutines is a fairly normal pattern.

~~~
tlrobinson
True, but the existing Node APIs don't do that, they return the error as an
argument to the callback.

~~~
snprbob86
The wiki page says:

    
    
        // these two lines are equivalent
        await a, b, c, d = moo(1, 2, 3);
        moo(1, 2, 3, function(a, b, c, d) {} );
    

So I don't see why not...

    
    
        // ...these two lines would be also equivalent
        await error, foo = bar(1, 2, 3);
        bar(1, 2, 3, function(error, foo) {} );

~~~
tlrobinson
Exactly. The error object is a return value of the "await"-ed function, not an
exception that's thrown.

Here's a concrete example. Normally in Node you do something like this:

    
    
      fs.readFile('/etc/passwd', function (err, data) {
        if (err) {
          // handle error
        }
        // normal processing
      });
    

The await version would look like this:

    
    
      await err, data = fs.readFile('/etc/passwd');
      if (err) {
        // handle error
      }
      // normal processing
    

Ideally it would look like this:

    
    
      try {
        await data = fs.readFile('/etc/passwd');
        // normal processing
      } catch (error) {
        // handle error
      }

~~~
koush
This is certainly something that is doable and crossed my mind, but I did not
want to have the await be restricted to functions that conform to the typical
return arguments. For now at least.

~~~
dbrock
How about

    
    
      try await foo = bar(baz)
    

as shorthand for

    
    
      await error, foo = bar(baz)
      if (error) throw error

------
weixiyen
I wrote a library (30 loc) to handle sequence and parallel flows.

Using only that, I rarely go over 80 character column limit that I impose on
myself. There is absolutely zero callback spaghetti whether it's 2 or 25
functions deep in the chain.

Tbh, callback spaghetti only happens to newer async programmers in the same
way that a newer programmer will write arrow code with if/else statements.

It's simply not a problem that needs to be addressed other than educating
people who are new to node.js with some example tutorials that use an async
helper library.

~~~
wladimir
So instead of spaghetti code (a tangled mess) it now is macaroni code (small
pieces of code everywhere, with unclear execution flow)? :-)

At least that's my experience with structuring asynchronous event-driven
programs (without coroutines).

~~~
n00kie
Yes :) Still waiting for a proper execSync (
<https://github.com/joyent/node/issues/1167> ).

Without it small build/utility scripts turn into macaroni cheese unless you
resort back to Python/Ruby/Bash - more languages, harder to maintain.

------
JonnieCache
This is why I'm planning on diving into Go when I eventually have a serious
realworld need for a lot of concurrency. None of this kind of mucking around
is necessary.

------
snprbob86
There was some discussion of adding this to CoffeeScript. The issues on GitHub
seem stalled. Would really love to see it happen soon!

~~~
artsrc
My understanding is that the CoffeeScript status is won't fix because the
transformation violates some constraints about the complexity of the mapping
that CoffeeScript values.

------
plasma
C#/.Net is planning on async/await features for its .Net 5 release:
[http://blogs.msdn.com/b/csharpfaq/archive/2010/10/28/async.a...](http://blogs.msdn.com/b/csharpfaq/archive/2010/10/28/async.aspx)

~~~
gtani
and in current release F# (current being pre-Build 2.0, i.e. RTM for over a
year, tho I can't comment on whether you'd really like to have VS Ultimate to
make it work)

<http://tomasp.net/blog/csharp-fsharp-async-intro.aspx>

<http://news.ycombinator.com/item?id=2999260>

<http://bvanderveen.com/a/owin-buffering-async/>

<http://devtalk.net/csharp/async-await-and-c-vnext/>

[http://msmvps.com/blogs/jon_skeet/archive/2010/10/29/initial...](http://msmvps.com/blogs/jon_skeet/archive/2010/10/29/initial-
thoughts-on-c-5-s-async-support.aspx)

(read comment)

~~~
thomasz
The current release is included in VS Professional, Ultimate isn't required.
It seems like the last recent installer for VS Shell is the CTP, so expect
some difficulties if you want to go the free route on windows, although it's
possible that SharpDevelop integrates with the latest source drop.

Anyway, it shouldn't be too difficult to build the compiler yourself and set
up a basic emacs/vim + mono environment for linux.

------
rjrodger
I think every competent programmer who comes to Node for the first time thinks
"hoo boy, better fix this callback stuff first", and immediately writes a
module to do just that. I know I did! :)

koush just took this to the next level.

But you know what. Just stop fighting the callback model. Adapt your coding
idioms and move on...

------
dualogy
In JavaScript callback spaghetti would kill me, but in CoffeeScript it's a
breeze and NOT something I want to "abstract away" at all for various reasons.
I suggest the solution to JavaScripts concoluted anonymous functions syntax is
CoffeeScript, or not inlining.

------
bluesnowmonkey
That "callback spaghetti" is called continuation-passing style.

"Programs can be automatically transformed from direct style to CPS." [1]

Do the math.

[1] <http://en.wikipedia.org/wiki/Continuation-passing_style>

~~~
thomasz
I don't fully understand why this makes callback hell better just because it
is known under a nicer name. CPS has it's use cases, but application code is
not one of them.

~~~
seabee
Specifically, CPS was intended as a mechanical program transformation to
remove the stack, not a style for programmers to actually write their programs
in.

------
voidr
You don't have to use inline functions all the time, you can do it like this:

    
    
      function foo () {.... }
      function bar () {....foo(); }
      function barfoo () {....bar(); }
    
      doSomething(barfoo);

~~~
jigs_up
And remember that 'bind' can save your life

this.io.sockets.on('connection', this.onConnection.bind(this));

------
kqueue
The proper fix is to introduce coroutines.

~~~
koush
That's exactly how this was done. I'm going to implement "yield" as well when
I have some time. They both just leverage coroutines.

~~~
tilgovi
node-fibers is one attempt to put coroutines into node. I like it because it
uses libcoro, which is nice and tight and uses ucontext, and because it works
as a module without forking the core of node. Once you've 'fiberized' the
server modules to start each request in its own fiber and implemented a call-
with-current-continuation on Function.prototype, we're pretty much at the same
place you've arrived but without new keywords.

------
damncabbage
How does this compare to <http://chris6f.com/synchronous-nodejs> ?

~~~
koush
Not sure to be honest! I am still very new the to scene so I am unaware of
this project. At first glance, we seem to have different approaches towards
the same goal though.

------
gfxmonk
Addressing the same problem (and more) in both node and the browser, via
compiling-JS-to-JS: <http://onilabs.com/stratifiedjs>

(disclaimer: I am the guy who worked on the now-defunct `defer` support in
Coffeescript, and am now working with the onilabs folk on the stratifiedJS
runtime)

------
zoips
This is great, but I'd rather not use a fork of Node, and Joyent isn't going
to integrate this, unfortunately. I wish that Google would just add yield to
V8, but they won't do that unless Apple adds it to whatever the hell their
interpreter is called now (Squirrelfish?).

------
jscheel
In the node.js project I just finished, I was also forced to write some
"async" c# for a separate system that the node.js project communicated with.
Now, I'll be the first to admit, I don't know c#... but, it was the first time
I had used both node.js AND c#. I have to say, the c# approach was a horrible
exercise in pain and agony. I started to run into significant nesting in
node.js, then realized it was because my approach was flawed. I abstracted
more, and I also started using async.js (<https://github.com/caolan/async>),
and everything was well in the world.

------
thesorrow
1) use async.js (classic javascript) 2) learn asynchronous patterns 3) profit

------
fleitz
It's a lot of syntactic sugar for one very specific type of monad. Why not
just patch the language to allow easy interoperation with any monad?

For example, F#'s let! binding works with any monad not just async.

~~~
DanWaterworth
I'm not sure monads work so well in languages without type-checking. That's
certainly my experience.

------
deepGem
This is very elegant. Just a couple of questions -

Using await - is the program flow suspended until the corresponding function
returns , or does await keyword act more like a 'pause and continue'
mechanism.

~~~
koush
It exits the function, and resumes it when the callback completes. So other
requests can be processed while the async function is (a)waiting.

~~~
deepGem
Perfect :).

------
plq
I like how the buzz around server-side javascript is progressing slowly from
"the next big thing!!!" to "okay, it has its shortcomings".

You will pry Python from my dead, cold hands :)

~~~
StavrosK
Seriously, it sounds like Python handles these sorts of cases much better,
with "yield". I wonder why everyone suddenly discovered JS for asynchronous
programming and rushed to it when Twisted has existed for years. Is it because
of V8?

~~~
baudehlo
It's partly because of the performance of V8 (my SMTP server, Haraka, is about
8-10 times faster than the Perl version I worked on, Qpsmtpd, for example),
and partly because of the "sync library" problem - with Perl (or Python) most
libraries that access things on the network or on disk are written
synchronously. That's not the case with Node - everything is assumed to be
async.

I've written a LOT of async code in Perl and C, and ran across this problem
countless times. It really is a blessing to not have to worry about it.

------
azoff
I definitely think the iterator case needs work:

<https://github.com/koush/node/issues/2>

------
robocat
Will Conant wrote a great article discussing three different libraries
(StratifiedJS, Streamline, node-fibers) to do with the problem:
[http://blog.willconant.com/post/7523275566/continuations-
in-...](http://blog.willconant.com/post/7523275566/continuations-in-node-js)

<http://tamejs.org/> also has a good write-up.

------
mattbillenstein
Man, just use python+gevent and be done with it...

------
swannodette
Changing JS is just not a great idea and I think Node.js has been wise to
avoid it.

If you're going to innovate, then design a language that compiles down to JS
that provides the innovation. CoffeeScript.

If you want to take that a step further, look at ClojureScript. Want delimited
continuations? Fine. All w/o requiring you to fork Node.js or CoffeeScript.

------
exclipy
It's great to see coroutines becoming more mainstream. I think every
programming language could do well with some way to abstract over the program
counter like this - ie. code that appears sequentially being executed
interleaved with other code (deterministically, unlike threading).

------
jeromeparadis
Very nice. I've also been using the async node.js module which is quite handy
for some other patterns. Of course, having direct support directly in node
would be the best. Can't wait for the yield keyword!

------
jeffz
Did anyone checkout Jscex? It basically has the same goal but implement as a
library. <https://github.com/JeffreyZhao/jscex>

------
jrockway
Or you can just do this:

    
    
       foo.step1 = function(){ do_some_thing(foo.step2) }
       foo.step2 = function(arg){ do_another_thing( ... ) }
       foo.step1()
       # enjoy

------
justatdotin
I don't accept that there's a "problem" that needs "fixing".

~~~
tilgovi
There's room for improvement. Most programmers shouldn't need to know how to
write asynchronous code just as they shouldn't need to know how to write
concurrent code. Consider mutexes vs Java's 'sychronized' vs transactional
memory: there's a way forward, and it's to reduce complexity for the average
programmer.

Remember that not everyone is as awesomely brilliant as you are.

~~~
justatdotin
Reduce complexity: sure. But if we have requirements that can be well defined
within an async FP paradigm, then maybe an async FP environment isn't such a
'problem'... and many of us "average programmers" already know lisp. If nodejs
doesn't suit everyone, probably not everyone should use nodejs. But if we're
going to have a go at it, let's not immediately dismiss node's natural style
as "an absolute mess"

------
jlongster
I've been wanting to implement something like this for a while! Great job. It
looks elegant and basically pushes the continuations into the background.

------
vessenes
koush, nice! I use your code every day when I reboot my phone, but I
appreciate this useful extension to node almost as much. :)

------
agentgt
It seems the async variables are just like Mozarts data flow variables

------
MostAwesomeDude
Perhaps it would be interesting to see how it was done in Twisted:
[http://twistedmatrix.com/documents/current/api/twisted.inter...](http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#inlineCallbacks)

We'd like to think that the C# guys were looking our way when they came up
with async/await, but there's no proof. :3

~~~
cheez
I love the way Twisted does it. Completely part of the way the language is
supposed to work. It's a work of art IMO.

------
diamondhead
JavaScript itself gives developers enough power to beautify messy async code,
already. Below is the rewrite of the first example in that page, with async
function composition:

====

/* <https://gist.github.com/1250314> */

var db = require('somedatabaseprovider'),

    
    
          compose = require('functools').compose;
    

compose.async(getApp, connect, select)({ url:'/price', host:'host', pass:'123'
}, function(error, shift){

    
    
      shift.conn.close();
    
      shift.res.send(shift.products[0]);
    

});

====

It's that easy to abstract those messy callbacks using some functional tools.

------
maxogden
theres this great thing called npmjs.org that is a place for stuff like this
if I am not mistaken

