
I love async, but I can't code like this - vamsee
http://groups.google.com/group/nodejs/browse_thread/thread/c334947643c80968?hl=en
======
kevingadd
The response by Isaac filled me with rage as soon as I got to 'No one actually
writes code like this'. The guy provides a real example from his application,
describes how alternative solutions stack up, and how he'd like to solve the
problem, and gets called out for presenting a 'strawman'. Sigh, newsgroups...

I personally share the original author's perspective on this and was
disappointed to see more sophisticated methods of writing concurrent code
(futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my
experience, chaining raw callbacks is too easy to get wrong for it to be
considered a reliable way of writing all your asynchronous logic, even if it
_is_ a great primitive to build a framework on top of. In particular, the
effort involved in propagating exceptions all the way through a callback chain
is rather significant, and making a mistake in a single function undermines
the reliability of the entire chain (usually resulting in work silently being
dropped on the floor). I wish more of an effort was made to address this
problem.

~~~
abp
You have seen what the author posted later on in the discussion?

<https://github.com/sam-mccall/node-plate>

~~~
DanielRibeiro
There are soo many interesting ways to solve this:
<http://news.ycombinator.com/item?id=2101724>

~~~
Sidnicious
Is that the wrong link, or are you threadjacking?

~~~
DanielRibeiro
Wow, wished I saw it earlier (can't edit now). Bad paste. Correct link:

[http://www.infoq.com/articles/surviving-asynchronous-
program...](http://www.infoq.com/articles/surviving-asynchronous-programming-
in-javascript)

------
teyc
The new C# 5.0 proposes to bring asynchronous keywords, which I believe works
like macros to turn imperative code into continuations.

    
    
      The “await” operator used twice in 
      that method does not mean “this method now blocks
      the current thread until the asynchronous 
      operation returns”. That would be making 
      the asynchronous operation back into a
      synchronous operation, which is precisely
      what we are attempting to avoid. Rather, 
      it means the opposite of that; it means
      “if the task we are awaiting has
      not yet completed then sign up the rest of
      this method as the continuation of that task,
      and then return to your caller immediately; 
      the task will invoke the continuation when 
      it completes.”
    

[http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/async...](http://blogs.msdn.com/b/ericlippert/archive/2010/10/29/asynchronous-
programming-in-c-5-0-part-two-whence-await.aspx)

    
    
        async void ArchiveDocuments(List<Url> urls)
        {
          Task archive = null;
          for(int i = 0; i < urls.Count; ++i)
          {
            var document = await FetchAsync(urls[i]);
            if (archive != null)
              await archive;
            archive = ArchiveAsync(document);
          }
        }
    

Microsoft has been very lucky to have someone like Anders Hejlsberg, who has
steadily brought a stream of features to an enterprise programming language.

~~~
contextfree
Note that this was (at least conceptually) derived from a (more or less)
equivalent F# feature that exists in the shipping version today.

~~~
gtani
backgrounder:

[http://www.infoq.com/news/2011/01/Why-
Async;jsessionid=512A2...](http://www.infoq.com/news/2011/01/Why-
Async;jsessionid=512A27F17C5A622A701F5DE128F3CB6B)

------
amadiver
`Bind` with named methods goes a long way to cleaning up nested callbacks.
Here's a (quick+dirty; I know I'm missing some `do`'s in there) version of the
same code in CoffeeScript:

    
    
      _ =
        fileMenu: ->
          mainWindow.menu "File", (e,f) => this.onFile(e,f)
    
        onFile: (err,file) -> 
          if err then throw err
          file.openMenu (e,m) => this.onOpenMenu(e,m)
    
        onOpenMenu: (err,menu) ->
          if err then throw err
          menu.item "Open", (e,i) => this.onOpen(e,i)
      
        onOpen: (err,item) ->
          if err then throw err
          mainWindow.getChild type('Window'), (e,d) => this.onDialog(e,d)
    
        onDialog: (err,dialog) ->
          if err then throw err
          #...
    
      _.fileMenu()
    

( `=>` is CoffeeScript for a function bound to the current scope )

~~~
flashingpumpkin
We've adopted a way that allows us to do basically this:

    
    
        mainWindow.menu "File", (err, file)->
            openMenu err, (err, menu)->
                getItem err, "Open", (err, item)->
                    click err, item, (err)->
                        getChild err, getChildType("Window"), (err, dialog)->
                            if err? and err instanceof NoFileError
                                # do stuff
                            if err? and err instanceof InvalidClickError
                                # do other stuff
                            if err?
                                throw err
                            # do stuff with dialog if no err
                            
    

Each function takes as first argument an error and as last a callback. If
there's an error, the function just passes the error up the callback stack
until it gets dealed with.

    
    
        fun = (err, args..., callback)
    

You can't pull that off if your code is heavily object oriented though:

    
    
        mainWindow.menu "File", (err, file)->
            # If mainWindow.menu errors, file is undefined 
            # and the call to openMenu throws an error
            file.openMenu err, (err, menu)-> 
                # ...
    

Breaking your code/api up with modules instead into classes fixes that.

------
jhpriestley
This is why I always find it sad that people are averse to learning about
"abstract, impractical" concepts like monads. Monads are abstract, but they
are far from impractical: they solve problems which you are already
encountering, and even already identifying as problems.

The best solution would be to add monadic syntax to javascript. You would get
CPS, coroutines, error propagation, early failure, and several other
capabilities all using the same syntax. Shit like generators or coroutines are
just ad-hoc special-cased versions.

~~~
swannodette
I'm completely convinced that the growing ubiquity of asynchronous and
multithreaded code will all drive us to embrace Object Oriented Programming
2.0 AKA Functional Programming. Can't come fast enough for me.

~~~
harryh
I, for one, hope that you are right (though the pessimist in me is skeptical).

------
Sephr
Using async.js ([http://eligrey.com/blog/post/pausing-javascript-with-
async-j...](http://eligrey.com/blog/post/pausing-javascript-with-async-js)),
you can do away with callbacks and still have asynchronous code by using the
yield statement. The following is an example (taken from my blog post) of
asynchronous code using async.js:

    
    
        yield to.request("/feedback", "POST", (
            yield to.prompt("What are your impressions of async.js?")
        ));
        yield to.inform("Thanks for your feedback!");
        // do more stuff here
    

The equivalent code with callbacks is as follows:

    
    
        async.prompt(
            ["What are your impressions of async.js?"],
            function (response) {
                async.request(
                    ["feedback", "POST", response],
                    function () {
                        async.inform(
                            ["Thanks for your feedback!"],
                            function () {
                                // do more stuff here
                            }
                        );
                    }
                );
            }
        );
    

It makes it very easy to handle DOM events, as demonstrated below.

    
    
        var clickEvent = yield myButton.next("click");
    

Full disclosure: I wrote async.js.

~~~
Groxx
Since you wrote it: any idea what browser support of Javascript 1.7 is at?
Wikipedia[1] currently thinks only Firefox has past 1.5, but I can't find
anything else that isn't ancient (in web years), and Wikipedia's tables of
things tend to get inaccurate pretty quickly.

[1] <http://en.wikipedia.org/wiki/JavaScript#Versions>

~~~
Sephr
Only Mozilla supports JavaScript (which is a mix of JavaScript-only features
and ECMAScript; it's a little complicated), and everyone else implements
ECMAScript. Some JavaScript features are in discussion of being added to
ECMAScript, but no current ECMAScript engines (e.g. V8) have implemented them
yet, barring the partial destructuring assignment implementation supported by
Opera's Futhark ECMAScript engine.

~~~
sid0
I think yield's coming to ECMAScript Harmony. That should simplify things a
lot.

------
olavk
There have been some different attempts to expand JavaScript syntax to make
this style of programming more elegant.

Narrative JavaScript <http://www.neilmix.com/narrativejs/doc/> has a single
construct, the arrow which turns everything after the arrow into a closure
which is passed as an argument.

StratifiedJS (<http://www.neilmix.com/narrativejs/doc/>) is more rich with a
number of new keywords: hold, spawn, wait etc. for asynchronous programming.

~~~
swannodette
Those are very cool projects, but the main issue is that their source
transformation is pretty aggressive. Until someone comes up with real, usable
debugging facilities for those, they're not going to catch on.

~~~
mpk
I recently attended a presentation on StratifiedJS and it is really cool, but
as you say they do some rather aggressive source transformation. You are
basically writing in a different language and loading in an interpreter in JS
to handle the new language. ObjectiveJ does the same thing. Take JS, extend it
and run it through your own interpreter. It still sort-of looks like JS, but
isn't.

Not only do you have the overhead of loading in an interpreter but you are
also left without good debugging facilities.

CoffeeScript doesn't have this problem because it compiles down to readable
JS, so if you're familiar with both CoffeeScript _and_ JS you can just use
native JS debugging tools to go at it. It does, however require the programmer
to be familiar with both languages, so it's usually not the right choice for
writing projects that you can hand off to another team at some later point.

~~~
afri
(Disclaimer: I work on StratifiedJS)

The point about not being able to use 'normal' JS debugging facilities with
StratifiedJS (or other 'higher-level' languages compiling to JS) is certainly
true, but IMO it is not nearly as bad as it sounds.

Debugging highly asynchronous programs (whether written in 'straight' JS or in
a higher-level language) with a 'normal' JS debugger isn't really much help:
the callstacks that the debugger spits out in no way correlate to the
asynchronous actions that are going on.

Having an abstraction that sits on top of JS, as it is the case with
StratifiedJS, gives us the opportunity to write a debugger that spits out
stack traces that actually reflect the true causal state of your program
logic.

Granted, we don't have such a debugger yet, but we will in the not-too-distant
future. What is already working is that StratifiedJS will amend exceptions
with the correct linenumber and module name of the original StratifiedJS
sources that threw the exception. So you don't have to manually correlate the
generated source code with the original SJS source code, like you would need
to do in systems that do a more or less 1-to-1 translation to JS.

------
lvh
It would be nice if people stop saying 'async' when they mean 'callbacks'.
Twisted's inlineCallbacks, Corotwine, Gevent... are all plenty async, and
there's not an explicit callback in sight. (I suppose you could argue that the
yield in inlineCallbacks and Monocle delimits a callback. You wouldn't be
wrong.)

~~~
wladimir
Of those I know gevent, which is indeed very nice, as it is 'implicitly'
async. It has the performance benefits of being async, but doesn't require you
to write callback-based macaroni network code.

~~~
lvh
Corotwine is exactly alike. Personally, I prefer inlineCallbacks. Explicit is
better than implicit... Basically I can say, okay, so, here something slow is
going to happen, and I'm willing to yield (hey look it even uses an
appropriate keyword) control over to the reactor. My main point is still this:
okay, yes, sometimes callback code can get ugly (with the apparent implicit
assumption plain old "blocking" code is always pretty...), but that has
nothing to do with the async backing.

~~~
wladimir
Plain old blocking code is not always pretty, but it is much more clear what
happens, and how errors are handled; all steps are in one place. Also, in my
experience, it is usually much shorter.

------
viraptor
Does anyone know if there are any plans for coroutines or similar mechanisms
in the new versions of ecma/javascript? Python's async frameworks like monocle
and diesel work around issues like that by returning another item from a
generator and receiving the result again, which works quite well and makes the
code look the same as the synchronous one.

~~~
grayrest
I know Brendan Eich has mentioned that he would like to bring python-like
generators into the language. They're in Mozilla's engine, were in ES4, and
they're a ES Harmony strawman.

<http://wiki.ecmascript.org/doku.php?id=strawman:generators>

Otherwise, I'm not really aware of anything. I know there's been efforts
around Promises but I think those have been restricted to libraries and I
think that even that's stalled because the CommonJS list is split between
Promises/A and Promises/B.

------
jamwt
Others have mentioned coroutines; just to avoid repeating a good conversation
that already happened on all this, here's the link:

<http://news.ycombinator.com/item?id=1549023>

------
nathansobo
JavaScript needs coroutines. They would solve a lot of problems.

~~~
icey
They won't be coming in node.js:
<http://twitter.com/#!/ryah/status/25685501193232384>

~~~
silentbicycle
But, why not?

I've done web programming in Lua (w/ WSAPI); Lua is generally like
Javascript's non-glue-huffing older brother, and coroutines/iterators remove a
LOT of issues.

I was working on an event-loop-based web server+framework in Lua, but then I
got into Erlang and realized how utterly lost the whole endeavor was. (But who
wants to program in Erlang? I hear it's bad at strings, or something.)

------
mpk
I really like JS and it has excellent closure support, but it's really obvious
that heavy async programming like we're doing nowadays with AJAX is not what
it was designed for.

The 'nested callback' problem is one I've encountered over and over. I've
figured out a relatively nice way to work around it. The solution goes
something like this.

\- Take the individual actions that you want and put them in closures.

\- These closures take 1 argument, which is always a callback function that
takes no arguments (this is the convention you structure everything around)

\- Put these closures in an array, in the order that you want them executed

\- Pass the array to a function that executes them for you (in Haskell it's
called a sequence)

I have a really quick and really dirty presentation about it here,
<http://www.moondust.dds.nl/text/async-presentation/>

Some code on github up here, <https://github.com/michiel/asynchelper-js> \-
the sequencer.js function is verbose, if you read the next SO link I've
included a three line example of implementing it.

And a stackoverflow answer using this code and some explanation here,
[http://stackoverflow.com/questions/4462605/jquery-ajax-
each-...](http://stackoverflow.com/questions/4462605/jquery-ajax-each-
callback-next-each-firing-before-ajax-completed/4462955#4462955)

And another one as a gist on github here, <https://gist.github.com/604730>
(for rendering a massive table from JSON data).

There's a lot more to this sequencer pattern. You can nest sequences, you can
have the sequencer run each callback using setTimeout(callback, 0) to release
control back to the main loop and get out of 'script is doing too much'
popups, etc.

Need to make an async call? Make a closure that performs the async call and
invoke the callback argument in your async callback.

Need to add some random JS code to a sequence? Wrap it in a closure and call
the callback when you're done. If the random JS code requires an async call,
see the previous paragraph.

Error handling, of course, is still awful. You can extend the sequencer
convention to add error closures as a second parameter, you can do everything
inside the closures in try/catch blocks and finally always call the callback,
etc. But that just tends to make a mess of things. So the examples just assume
everything is going fine.

It's not an ideal solution, but it's native JS and works _everywhere_.

It allows you to structure your code into functional blocks and work on them
separately instead of making everything one big nested callback mess.

------
sam-mccall
Sorry I missed the discussion!

Just wanted to add a pointer to what I ended up using:
<https://github.com/sam-mccall/ship>

It's lets you create pseudo-threads, you pretend you're calling a sync
function, and get a promise back, which you can call methods on (giving more
promises) etc.

It needs a rewrite to allow objects in different pseudo-threads to interact -
i.e. the threads to cross. Currently it's just a queue of actions to perform
which doesn't allow this.

------
uriel
This is one of the reasons I love CSP and languages with CSP-like concurrency
like Go (and Erlang, although it is more CSP-inspired than CSP-like).

I really can't understand why anyone would want to use anything else for
concurrent programming.

------
moomin
I can't help feeling that all of the examples contain too much copy and
pasting. This is dealt with by taking a higher order approach to the problem.
Which is what async.js is for.

    
    
       doSomething = (c) ->
          async.waterfall [
             (c) -> mainWindow.menu "File", c
             (file, c) -> file.openMenu c
             (menu, c) -> menu.item "Open", c
             (item, c) -> item.click c
             (c) -> mainWindow.getChild type("Window"), c
             (dialog, c) -> dialog.doSomething c null
          ]
    

It's not perfect, but I think it's pretty readable and succinct.

------
swannodette
Perhaps what we need is something like CJSR (Common JavaScript Runtime).
Languages that compile to JavaScript generate standardized debugging
information as JSON (CoffeeScript has been pushing for this). These languages
can then use a shared set of in-browser debugging tools (JSDB?). The fact that
Safari, Chrome, and FF all support common browser extension architectures as
this makes this solution even more compelling.

Agressive source transformation becomes less of an issue.

------
mmaunder
I tend to use Node for specific tasks like running a process that may take
from 0 to 20 seconds without having a thread block. Using it for discrete
tasks that it's suited for and then doing the bulk of my programming on
regular vanilla synchronous server processes has worked for me as a way to
manage complexity.

Until a standard model of async emerges I think programming large web apps on
node will feel a little like hiking with a stone in your shoe.

------
Kilimanjaro
There is a huge gap for a server side implementation of JS and node is not the
one the masses are asking for, despite all the hype surrounding it.

Just take python, ruby or php, and make JS on the server on par with them.
Forget about creating servers, async stuff and coroutines, 90% of moms and
pops websites don't need them.

There is a huge momentum behind SSJS but it is all being wasted waiting for
node to be the mother of all js apps. It will never be.

------
EGreg
I felt the same way, but thankfully Javascript lets you write really cool
things that handle the flow of your code in ways that correspond to the way
you think of it.

I encourage you to check out <http://news.ycombinator.com/item?id=2101789> ...
with those functions javascript will be pleasant again.

------
statictype
I think the most likely solution would come from a language extension to
something like Coffeescript.

It could work similar to the async monad in F#:

#asnc x = () -> var f = window.menu("file"); var m = f.openmenu(); m.click();

would unwind as:

window.menu("file",function(f) { f.open(function(m) { m.click(); }); });

------
nikils
isn't arrows [http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-
and...](http://www.cs.umd.edu/projects/PL/arrowlets/example-drag-and-
drop.xhtml) supposed to solve this problem.

------
fleitz
Use F# instead

    
    
      let async_copy (in:Stream) (out:Stream) =     asnyc {
          let buf = Array.create 8192
          let rec copy = async {
              return! match in.ReadAsync(0,buf.Length,buf) with
                      | 0 -> ()
                      | x -> do! out.WriteAsnyc(0,x,buf); 
                             return! copy; }
          return! copy; }
    

There's an async copy routine, not much more overhead than a sync copy.

    
    
      let sync_copy (in:Stream) (out:Stream) = 
          let buf = Array.create 8192
          let rec copy = 
             match in.Read(0,buf.Length,buf) with
             | 0 -> ()
             | x -> out.Write(0,x,buf); 
                    copy    
          copy

~~~
icey
Slight typo in the first line of your code; asnyc should be async.

Line 6, WriteAsnyc should be WriteAsync

The intentions were very clear, just posting this in case someone tries to
copy & paste it.

------
locopati
Is there a reason that events don't work in the situation? The async result
handler fires an event 'getXFinished' and any function that needs to respond
does so. When they are finished, they fire events. And additional functions
are called. And so on.

