
Managing Node.js Callback Hell with Promises, Generators and Other Approaches - jguerrero
http://strongloop.com/strongblog/node-js-callback-hell-promises-generators/
======
tgriesser
Disappointing that most promise articles/tutorials reference Q as the
preferred promise library, rather than bluebird -
[https://github.com/petkaantonov/bluebird](https://github.com/petkaantonov/bluebird)

Bluebird is not only so fast that it's almost on par with callbacks[1], but
also innovates on the ability to filter on catching different error types[2],
support for generators[3], context binding[4] among other things.

[1]:
[https://github.com/petkaantonov/bluebird/blob/master/benchma...](https://github.com/petkaantonov/bluebird/blob/master/benchmark/stats/latest.md)

[2]:
[https://github.com/petkaantonov/bluebird/blob/master/API.md#...](https://github.com/petkaantonov/bluebird/blob/master/API.md#catchfunction-
errorclassfunction-predicate-function-handler---promise)

[3]:
[https://github.com/petkaantonov/bluebird/blob/master/API.md#...](https://github.com/petkaantonov/bluebird/blob/master/API.md#generators)

[4]:
[https://github.com/petkaantonov/bluebird/blob/master/API.md#...](https://github.com/petkaantonov/bluebird/blob/master/API.md#promisebinddynamic-
thisarg---promise)

~~~
angersock
We _just_ started using Q for our all--aaaargh, motherfucker.

Great to see progress being made, I guess. :(

~~~
tlrobinson
The important thing is you're using promises. The differences between Q and
Bluebird are minimal, compared to promises vs raw callbacks.

I just found this after some cursory Googling and haven't tried it at all, but
it appears to be a compatibility layer for Q on top of Bluebird (written by
the author of Bluebird):
[https://gist.github.com/petkaantonov/8363789](https://gist.github.com/petkaantonov/8363789)

~~~
esailija
I would not say this[1] is a minimal difference :P

[1]:
[https://github.com/petkaantonov/bluebird/blob/master/benchma...](https://github.com/petkaantonov/bluebird/blob/master/benchmark/stats/latest.md)

~~~
saurik
I am pretty certain he means in terms of code delta.

------
rdtsc
Looking at it, in general, ignoring that I know about Node internals, you know
what is the simplest way to solve this "problem"?

It is something like this:

\---

    
    
        callback(find_largest(get_stats(read_files(dir)))
    

\---

1 line.

If an error happens, an exception should be raised.

The fact that there are multiple such requests happening or that some of those
operations have to wait for a select or epoll loop to return, should be
handled by an underlying framework. The fact that even with promises you have
to write 27 lines of code to do that, should be making you worried.

~~~
inglor
Please stop spreading FUD.

With promises (Bluebird) this sounds like very little code in practice,
assuming promisifyAll has been called the FS module:

\- One line to read all files in a directory

\- Map them with .map to a .stat call on them, returning array of promises.

\- Reduce them to the largest element.

Something like:

    
    
        var fs = Promise.promisifyAll(require("fs"));
        var largest =  fs.readdirAsync("/your/file/path").map(function(file){
             return fs.statAsync(file);
        }).reduce(function(x,y){
            return (x.size > y.size) ? x : y;
        });
    

With arrow syntax:

    
    
        var largest = fs.readdirAsync("/your/file/path").
                      map((file) => fs.statAsync(file)).
                      reduce((x,y) => (x.size > y.size) ? x : y);
    

Alternatively, with Generators using Bluebird promises

    
    
         var largest = Promise.coroutine(function *largest(dirname){
            var files = yield fs.readdirAsync(dirname);
            var sizes = yield files.map(name => fs.statAsync(name));
            return files.reduce((x,y) => x.size > y.size ? x : y);
        });
    

If one error occurs, the promise rejects. VERY far from 27 lines of code. Not
to mention you can of course compose these in one line if you extract them to
methods like in your code:

    
    
        var largest = fs.readdirAsync("/your/file/path").map(fs.statAsync).reduce(largest);

~~~
rdtsc
> Please stop spreading FUD.

Was that addressed to strongloop.com? They are the ones that posted an example
with promises using 27+/-whitespace lines of code and the initial problem
algorithm only had 4 lines.

Sounds like you should write a blog post and not them about using promises.

I don't use Node.js on daily basis and am not familiar with promises so just
commented as an outsider based on the article.

Your last one line looks fantastic and makes it clear what is going on. Any
reason you can think of why that wasn't the "obvious" solution to the original
posters of the article?

~~~
inglor
rtdsc: > Was that addressed to strongloop.com? They are the ones that posted
an example with promises using 27+/-whitespace lines of code and the initial
problem algorithm only had 4 lines.

Oh wow, this is my fault, I didn't read what they did there. I have __no
idea__ why it took them 27 lines to implement this. It's rather trivial with
promises.

\- They have a .then followed by a .map there causing nesting for no reason
whatsoever.

\- They also get the file name, but handle this very poorly (with an array),
where they could just do it as a part of the .map (the .all and .then are
completely redundant) there

\- They assume the possibility of directories, I didn't. That .filter is
justified but that's the only thing one might have to add here.

\- Generally, it's poor promise code.

I guess that promises are just not their tools of choice and they just used
them _for that post_.

------
shtylman
The generator example has no error handling (at least not from what I can
see).

Once the "callback hell" example was re-written with named functions, it is
not much different than the generator example and depending on how you are
testing, easier to test because it separates out IO from logic (Which you
could still do in the generator example).

One thing callback style code has begun to make me think about is the
nesting/IO complexity of the functions a I write. When you nest, the IO/event
boundary is visible in the code (by the indentation). As you continue to nest,
you can start to get a clear idea of the amount of IO being done and where a
particular function may be doing too much. Anyhow, most of this is preference
anyway :)

~~~
sync
A simple try/catch around the whole thing would take care of error handling
for the generator example.

~~~
inglor
only until you have a syntax error :)

------
pak
As you read all these many lines of code to simply find the largest file in a
directory, recall that in any language with friendlier blocking/non-blocking
semantics, it could be as simple as (e.g. in Ruby)

    
    
        Dir.glob('*').select{|f| File.file?(f) }.max_by{|f| File.size(f) }
    

Plus, the error semantics of this code are sane: if something goes wrong I get
an exception I can rescue. If there were no files in the directory, I get nil.

Statting a bunch of files is rarely so costly that parallelism is warranted,
unless it is impressed upon you as in Node. The cost in readability and
dreaming up generators and promises to solve such a mundane problem is likely
much worse. Before you charge headlong into building your next webapp with
Node, consider this article food for thought.

~~~
rmgraham
It's definitely a trade-off. What Node gives you is async when you need it at
the cost of giving you async even when you don't.

The pain and constraints introduced by this forced use of async sure has
spawned a lot of wheels of all sizes, colours, and compositions.

~~~
jerf
Ruby is the wrong comparison. Write that same code in Go, suitably translated,
and it _would_ be fully "async".

I hit Go just because it's the most Algol-ish of the sane languages. There's
several other options where you just _write the code_ and the compiler and
runtime do the async for you.

------
programminggeek
I remember doing a deep dive into node and evented JS a couple years ago and I
realized that I couldn't recommend it to my team because the code would become
such a huge mess on an large system that whatever performance you gain, you
lose in maintainability in a large team environment.

The best I could find at the time is IcedCoffeescript, which sort of replaces
the whole existing coffee script tooling, so now you are dealing with custom
tooling to install for each team member as well.

I find it sad that years later it looks like what IcedCoffeescript did is
still the cleanest looking solution to the problem. It feels like the evented
model is maybe great for smaller things, but horrible for building and
reasoning about in larger systems on larger teams.

------
DonPellegrino
I'd like to mention my favorite tool to deal with Callback Hell: Streamline, a
JavaScript (and CoffeeScript) language extension that can compile to either 1)
callback spaghetti 2) Node-fibers or 3) generator code. It basically writes
the mess for you. It lets you focus on the higher order concepts and provides
configurable parallel array operations as well as many other convenient
features. The callback "error parameter" is thrown as an exception to be
caught with a basic try/catch. I highly recommend that any Node and JS
enthusiast give it a try.

[https://github.com/sage/streamlinejs](https://github.com/sage/streamlinejs)

------
porphyry3
There is another nice feature of ES6 that helps a lot with callbacks and
promises: __proxies __. Check out this article I wrote on Flippin Awesome
([http://flippinawesome.org/2013/11/18/taming-asynchronous-
jav...](http://flippinawesome.org/2013/11/18/taming-asynchronous-javascript-
programming-with-ecmascript6/)). The following is some sample code to show how
code can be simplified with ES6 proxies, making async functions chainable
(example in coffeescript):

    
    
        getUser = (user) ->
            _("https://npmjs.org/~#{user}")
              .get()
              .extractLinks()
              .filter( -> /package/.test(arguments[0]) )
              .map( -> "https://npmjs.org#{arguments[0]}" )
              .log()
        
        getUser("foo")
    

where `get` is THE jQuery get for Ajax requests returning a promise and is
chained with the rest of the functions in a pipeline fashion. I've found that
these techniques can work very well for form validation, as [you can check
here]([http://www.vittoriozaccaria.net/chained/demo.html](http://www.vittoriozaccaria.net/chained/demo.html))
— Firefox required.

Edit: I wrote in markdown but it seems that only verbatim is supported. Edit2:
some clarification

------
pron
The problem with all of these solutions (callbacks, promises, etc.) is that
they make you adopt a certain programming style not because it's the most
appropriate from a design perspective, but rather to work around scheduling
limitations.

Lightweight threads first remove the problem, and then let you choose the most
appropriate programming style for your domain. See
[https://news.ycombinator.com/item?id=7179144](https://news.ycombinator.com/item?id=7179144)

------
campbellmorgan
an important omission here is the async.auto function which is a phenomenal
relatively new addition that takes a amd approach to asynchronously running
multiple tasks at the same time:
[https://github.com/caolan/async#auto](https://github.com/caolan/async#auto)
if you haven't seen it already

------
ilaksh
Here is how I would do it using ToffeeScript
([https://github.com/jiangmiao/toffee-
script](https://github.com/jiangmiao/toffee-script))

    
    
        fs = require 'fs'
        async = require 'async'
        path = require 'path' 
         
        module.exports = (dir, cb) ->
          err, files = fs.readdir! dir
          paths = (path.join dir, file for file in files)
          er, stats = async.map! paths, fs.stat
          largest = 0
          for stat, i in stats
            if stat.isFile and stat.size > largest
              fname = files[i]
              largest = stat.size
          cb fname
    

I use ToffeeScript for everything now, since it makes things much easier to
read.

------
shubhra51
Promises have always been debatable. The new Generators is probably what I am
looking for.

~~~
onestone
Generators and Promises play very well together.

------
inglor
Why would one use Q promises in Node in 2014? They're very very very slow.
Especially when there are alternatives like Bluebird promises that are almost
as fast as callbacks.

------
pekk
When we had these things in other languages like Python, node's approach was
better. Now that node has adopted them, node is even better still, I guess.

~~~
davidw
Some other camps ( [http://www.erlang.org/](http://www.erlang.org/) ) still
strongly prefer their own approach, and I can't say I blame them.

------
gfodor
This is satire, right?

------
ksherlock
All that work and the findLargest routine doesn't seem to benefit whatsoever
from fs.readdir or fs.stat being asynchronous.

