
Why I am switching to promises - shawndumas
http://spion.github.io/posts/why-i-am-switching-to-promises.html
======
beering
Oh goodness... if only there was a way for the computer to manage all the
bookkeeping of what happens after what, in a way where multiple things could
run at approximately the same time, without the programmer having to thread
"promises" or "callbacks" through every single piece of non-trivial code.

We could even give a name to these sequences of operations... maybe call them
"strands"... or "filaments"...

Using libraries in Node.js seems to be a matter of 1) finding a library, 2)
taking a hammer and smashing it into your existing choice of continuation-
tracking mechanism. We end up with N x M different ways of using a package,
depending on what style the package author chose vs what style the programmer
chooses. "nodeify" is a bandage on top of a missing limb.

Do people really put up with this? Just look at how many different things were
benchmarked in this article---I counted 21. How many more ways of reinventing
basic programming-language design is the JS community going to come up with
before we realize that the emperor has no clothes?

~~~
gcr
(Edit: Oops! Sorry, I see that you're already familiar with continuations, but
let me explain them for those who aren't)

Sometimes, fundamental switches in thinking are worth it. :)

For example, continuations would be _perfect_ here, and are a wonderful tool
for simplifying "nested callback hell" type problems.

But why should a web developer care about continuations?

Consider Paul Graham's Arc challenge: Build a web page that presents a form to
the user. After they submit the form, a new page presents them with a link.
The link takes them to a third page where the submitted value is presented
back to the user. The restriction is that the input field must not be passed
in via the URL.

Right now, the user must write in "continuation-passing style", which means
threading callbacks, like this:

    
    
       function serve(req,res) {
          askUserForValue(req,res, function(value){
              showLink(req, res, function(){
                  render(res, {"user-value": value});
              });
          });
       }
    

Or perhaps the user could somehow stuff the variable into a "session" library
with three separate URL routes:

    
    
        function serve(req,res) {
           render the form with the POST action of /submit;
        }
    
        function submit(req,res, value) {
           req.session["uservalue"] = value;
           render the link that sends the user to /view;
        }
    
        function view(req,res) {
           render the value of req.session["uservaule"];
        }
    

This one is even more fragmented since you have three separate functions that
model one user interaction flow! I certainly like it less than the callback
example -- at least I can keep one in my head.

We can do better though. Continuations let you write this:

    
    
       function serve(req, res) {
          var value = askUserForValue();
          showLink();
          render(res, {"user-value": value});
       }
    

If node.js supported continuations, the askUserForValue() function would
respond with the form and _suspend_ the serve function right then and there.
After the user made a new HTTP call, the suspended askUserForValue() call
would return to serve() and its return value would be placed into our
variable. Then, the showLink() function could render the link and _suspend_
the function again. Once the user makes another HTTP request, the function
resumes, and so on.

The server would have to store the _state of the computation itself_ to disk
so it could _continue_ later.

With continuations, we can pretend the user's web browser is the program
counter. This frees my mind up. First, I don't have to keep callbacks in my
head since there aren't any callbacks anymore. Second, stack traces work the
way I expect because functions explicitly call each other. Third, if
askForValue() throws an exception if the user inputs a string where I entered
a number, no problem! Just wrap it in a try{} block that asks the user to
enter a valid value, _as if I 'm writing a text program that calls
readline()._

Also note that this is different from just letting each user interaction be in
its own thread! Consider:

1\. Continuations should be serializable to disk. You wouldn't want blocked
threads to just pile up as more users visit your website, and if your server
crashes, you certainly wouldn't want to forget their state.

2\. The browser "back" button would have to _step a program backwards in time_
; something that is unsupported in a naïve thread-equals-user-interaction
system. (E.g. how many of you have booked an airplane ticket with
Travelocity/Expedia/whatever, pressed "Back", and then got a big error because
the server didn't know how to step backwards through its user stack?) But the
ability to save continuations to disk is the ability to save the current state
of the program to disk, which makes stepping back in time easy: you can simply
reload what was about to happen, just like savestates in a video game
emulator.

This is not a new idea. Many Common Lisp and Smalltack web frameworks work
well with continuations and have done so for years (the first Seaside release
was 2002, I'm not kidding you!)

Racket, a great dialect of Scheme, ships with such a web framework out of the
box. Here's my implementation of the Arc challenge in Racket. Note how the
'start' function is implemented:
[https://gist.github.com/gcr/ad116f565e105c8b2e0d](https://gist.github.com/gcr/ad116f565e105c8b2e0d)

It's super easy to try this out and you don't need to fiddle with dependencies
or anything. Just download Racket from [http://racket-
lang.org/](http://racket-lang.org/) , copy+paste the code into DrRacket's
giant text box on the top, press Run, and your web browser will pop up with
the example.

~~~
spion
This is not possible with promises or generators, because their flow is
unidirectional (so no back button). It may be possible to come up with an
abstraction that works like promises (.then chaining) but can be rewinded, in
which case the code would look like this:

    
    
      askUserForValue
        .then(value => res.showLink()
          .then(_ => res.render(res, {"user-value": value}))
    

Infact I think something like this could probably be built with monadic
observables and a nice library. The only thing you couldn't do is
automagically keep intermediate state anywhere else other than memory :/

The thing is, isn't this just a pipe dream? The moment you need to interact
with an external system, like say an SQL database, you can say good-bye to
rewinding, or be fine with the fact that you might commit a transaction twice,
or otherwise come up with a complex method to also rewind the database state,
or write your own database that supports this :)

Edit: Also, these days this approach is largely unnecessary. Now the entire
thing would work on the client and the server would only be responsible to
provide an API :P

~~~
gcr
Your last paragraph is why Racket provides ways of invalidating "in-progress"
continuations. Ideally, in any user flow, you'd want to have the only side-
effect operation occur at the end of the flow (User clicks the 'Book ticket'
or 'Process transaction' button); the hard part is usually modeling it
beforehand in a way that allows the user to navigate naturally back and forth
between the parts of the application/open it up in new tabs or whatever.

But sure, there are limitations of course. :)

~~~
spion
I see. It definitely looks powerful. Do you know if this is possible in any
non-LISP language?

I mean, invalidating of the "observable" chain could also be done in JS, but I
don't think that I know of other languages that could automatically transfer
the state to the client.

~~~
gcr
I'm not sure. I honestly think it's a culture thing. Many Lisp communities
embrace continuations, but everyone else rightly shies away because they make
your head hurt.

Ruby(!) supports them, so it might be possible for Rails to create something
like the pattern I mentioned. In fact, Ruby people use continuations for a few
things like the built-in Generator class and making "restartable exceptions"
so the thrower can figure out how to deal with the problem rather than the
catcher, which is sometimes quite useful:
[http://chneukirchen.org/blog/archive/2005/03/restartable-
exc...](http://chneukirchen.org/blog/archive/2005/03/restartable-
exceptions.html)

Properly implementing continuations is a huge implementiation burden though,
which is another reason why the're uncommon. The language would have to make
the stack frame itself slicable and serializable, and it's hard to get good
performance when the call stack itself becomes a "first class" language
construct. Consider a duplicate() function that saves the state of a program
and then spawns ten threads that each reload that state! Have fun implementing
continuations in a way that supports that, Guido! :)

------
pornel
Promises have two awesome things:

* With callbacks _interface_ of the function hardcodes whether the function is strictly synchronous or may use _any_ async code. Change of one little helper function from sync to async may have ripple effect throughout the entire codebase (you have to add callback to all its callers, and all their callers, and their callers...).

Promises fix that. Sync and async functions take arguments and return values
the same way, so when you change sync to async you only need to change source
and consumer of the value, and nothing in between (e.g. same code can cache
results of (memoize) promise-returning functions and sync functions).

* With ES6 yield async calls look and behave almost like sync calls. No callbacks. You don't have to use weird error handling schemes — just use try/catch! You don't have to have a library to iterate over things — just use for()!

A call that used to look like:

    
    
        use(compute(arg))
    

was twisted by async to be:

    
    
        compute(arg, use)
    

Promises change it to:

    
    
        compute(arg).then(use)
    

and yield changes it back to nearly original form:

    
    
        use(yield compute(arg))

------
nadaviv
> if (err) return callback(err)

> That line is haunting me in my dreams now. What happened to the DRY
> principle?
    
    
      iferr = (succfn, errfn) -> (err, a...) ->
        if err? then errfn err else succfn a...
    
      show_user = (id, cb) ->
        db.query ..., iferr cb, (user) ->
          ...
    

FTFY, no promises needed. (not to say that promises don't have other
advantages, but that specific complaint is easily fixable with an higher-order
function to abstract it away)

(JavaScript version:
[http://coffeescript.org/#try:%20%20iferr%20%3D%20(succfn%2C%...](http://coffeescript.org/#try:%20%20iferr%20%3D%20\(succfn%2C%20errfn\)%20-%3E%20\(err%2C%20a...\)%20-%3E%0A%20%20%20%20if%20err%3F%20then%20errfn%20err%20else%20succfn%20a...%0A%0A%20%20show_user%20%3D%20\(id%2C%20cb\)%20-%3E%0A%20%20%20%20db.query%20foo%2C%20iferr%20cb%2C%20\(user\)%20-%3E%0A%20%20%20%20%20%20bar))

~~~
spion
No its not. I address this immediately afterwards (below "But spion, why don't
you wrap your calbacks?"), but basically, this simple wrapper makes stack
traces unusable.

Bluebird (and also Q) stitches together stack traces from multiple previous
events. This is amazingly awesome.

------
moron4hire
I see stuff like:

    
    
        promise.when(function(value){
           //blah blah blah
        })
        .error(function(err){
           //handle it
        })
    

and I think to myself I'm seeing text replacements:

    
    
        "promise.when(function(value)" = "try"
        ".error(function" = "catch"
        "})" = "}"
    

So how is this actually different from the code we're supposedly too lazy to
write or too stupid to figure out or something in the first place, thus
requiring this slightly more verbose form?

And where is this all going? Like, what's the right granularity of lines-of-
code to when chains? If programmers are too lazy to try/catch, then why aren't
we too lazy to when/then/error a gigantic block, or you know, not at all? And
if we were to instead [w|t]hen/error every line, then I think about DRY and
and it occurs to me the most likely next step is a language that "implicitly
whens your code", that every line is written like regular procedural code, but
automagically has "when" inserted in the middle.

Or in other words, you know, continuations.

But it occurs to me that the only reason any of this stuff is of any interest
is because JavaScript, and therefore Node, _still_ lacks real MT support, so
you have to jump through these infinitely fractal spilling callback tangos to
fake it.

Web Workers was 2009, people. 5 YEARS MAN. Five years that we've been waiting
to hear anything about having sane primitives for MT.

~~~
spion
Nope, `doStuff().then(function(value)" != try`

A closer representation would be `try { var value = doStuff()` which has
similar verbosity.

ES6 will have generators and "arrow" function syntax. Meaning:

    
    
      doThing().then(function(value){
         //blah blah blah
      })
      .catch(function(err){
         //handle it
      })
    

becomes either:

    
    
      try { 
        var value = yield doThing();
        //blah blah blah
      } catch (err) {
        //handle it
      }
    

or

    
    
      doThing().then(value => /* blah blah blah */)
        .catch(err => /* handle it */);
    

meaning you can pick your paradigm :) Except that the functional one composes
slightly better, perhaps.

    
    
      doThing().then(value => /* blah blah blah */)
        .catch(only(ParseError, err => /* handle it */));
    

vs

    
    
      try { 
        var value = yield doThing();
        //blah blah blah
      } catch (err) {
        only(err, ParseError); // argument repetition
        //handle it
      }

~~~
moron4hire
Uhhh, you're not convincing me that you've said anything meaningful towards
improving software quality.

~~~
spion
As opposed to what, threads? They have contributed towards improving software
quality? In most languages with mutable state they've been a disaster --
correct me if I'm wrong.

Adding threads in JavaScript would also be a disaster. Practically everything
is mutable and there are absolutely no locking mechanisms.

~~~
audreyt
One possible solution to introduce threads to JS is to isolate threads from
each other and use only message-passing:

[https://github.com/audreyt/node-webworker-
threads/](https://github.com/audreyt/node-webworker-threads/)

~~~
spion
Sorry, but share-nothing threads that can only communicate via (afaics)
serialized message passing are no longer threads. There is very little benefit
to those compared to just spawning child processes.

------
ilaksh
I hope that everyone who writes JavaScript will do themselves a favor and try
out ToffeeScript. This offshoot of CoffeeScript simplifies things. For
example:

    
    
        if fs.exists! thefile
          e, datastr = fs.readFile! thefile, 'utf8'
        else
          datastr = 'not found'
        success = db.update! 'somekey', datastr
        res.end success
    

So in that example even though there are three callbacks it looks like a
normal synchronous flow.

~~~
lucaspiller
I haven't heard of ToffeeScript before, but this is pretty much how Erlang
works. Whenever you make a blocking call, all of the async stuff happens under
the hood, so that you just deal with sequential code.

Erlang's concurrency model deals with lots of lightweight processes. Rather
than having callbacks it uses a message system. Whenever a process waits for a
message it yields so other processes can be executed.

In your example there would be a process separate from the main[0] process
that handles the database connection. When you request the update to the
database, you are just sending a message to the database connection process.
This will receive the message and handle the update when it can[1], and once
complete, send you a message back with the result of the update. During the
waiting period the VM will continue executing other processes. No callbacks
needed :)

[0] Erlang concurrency is based upon the actor model. There isn't really a
main process, but for the sake of explanation...

[1] The database connection process will actually block while making the
update, so if you have 10,000 processes calling this, you will have a
bottleneck. As such you should have a pool of database connection processes,
rather than just one.

~~~
ilaksh
Yeah Erlang looks awesome and I should definitely try to learn it at some
point.

To me Node.js and ToffeeScript seem a lot more straightforward and have the
advantage of the whole package system which I believe is more advanced, as
well as the npm ecosystem with 54,000 packages.

But anyway it would be good to actually learn Erlang and use it in some
projects so that I can really evaluate it.

I'm not sure but I think it might be a lot more resource efficient for one
thing.

------
rtpg
Slightly off topic, but I've been working in Haskell, and recently realized
how the laziness of the language means everything is basically asynchronous by
default, but you can still write your code in a synchronous fashion. No real
need for promises or callbacks because lazy values _are_ the promises. Well,
when using lazy IO, at least.

I'd actually much prefer lazy values be put into Javscript than almost any
other feature (real module support is another big one).

~~~
rapala
Then again, Simon Peyton Jones has said that were he to make a new Haskell, it
would have strict evaluation. As far as I know, lazy IO is considered a pain
in the butt. That beeing said, laziness does have its merits and some ideas
can be elegantly stated in a lazy language.

~~~
rtpg
do you have a source on this? For me laziness is the single strongest thing
going for Haskell.

~~~
rapala
It's from a retrospective on Haskell: www.cs.nott.ac.uk/~gmh/appsem-
slides/peytonjones.ppt

There seems to be two arguments against laziness. Firstly, it makes reasoning
about performance hard. Secondly, forcing strict evaluation is tricky, maybe
harder than it would be to do optional lazy evaluation in a strict language.

------
eldude
The problem with discussions of promises vs ... is there's usually a great
deal of misunderstanding about what problem promises are solving: enforcement
of the callback contract. I could go on for hours about the subtleties in the
conversation, but the gist of it boils down to this:

    
    
        Node callbacks have a strict but entirely unenforced (in userland) contract:
    

The contract:

    
    
        * Callbacks should be called OAOO
    
        * Never throw (explicitly or implicitly) after origin tick (aka pass post-tick errors to callback)
    

There is also an increasingly common informal contract:

    
    
        * Never throw ever (aka pass ALL errors to the callback)
    

But there are a couple of issues that get in the way of successful enforcement
of this contract:

    
    
        * When external libraries violate this contract (library A calls library B,
          but library B violates the contract in a way that makes it difficult for
          library A to easily maintain the contract)
    
        * Asynchronous errors
    
        * Crash-worthy exceptions
    

Inevitably, this all continually comes back to error handling over and over
again, with the core issue being that node.js has 2 divergent methods for
error handling: exceptions (try/catch/throw) and error passing
(callback(err)). Exceptions requires opt-in error handling (crash by default)
while error passing requires opt-out error handling (crash on demand) "and
never the twain shall meet."

Promises attempt to resolve this by coercing all exceptions to error passing,
but not all exceptions are safe to be passed. Specifically, any exception
originating from core that is not an invalid argument exception (think 4XX vs
5XX class errors) cannot safely be caught / coerced / continued on. Also, not
all errors passed to callbacks are even theoretically passable as the
unofficial policy of core is that core callback errors are only
distinguishable from exceptions in that they occur after the origin tick. In
other words, they are equally crash-worthy and equally non-crash-worthy.

So promises are an improvement because when used pervasively, they enforce the
formal and informal callback contracts, while providing a continuable-like
representation of the value that can be passed around.

 __This benefit of contract enforcement is entirely independent of the
specific API implementations. __

The stepup[1] library trivially accomplishes contract enforcement by
integrating the async trycatch[2] module without using promises or requiring
pervasive usage. I 've used them both for years now in various professional
projects, with trycatch in use here at LinkedIn. Additionally, stepup could
easily return a continuable to support a passable value representation.

Async generators will solve most of this though, allowing node.js to ditch the
error passing for exception handling, but since Error subclassing and typed
catches are basically not supported in node.js the language keeps getting in
the way of a satisfactory complete solution. This is a whole other discussion.

In conclusion, error handling in node.js is an undesigned mess of which the
core contributors don't even seem to fully understand or be aware[3,4]. FWIW,
node.js' crash on error design is a DoS liability, which spion addresses in
the article, and which LinkedIn uses trycatch to avoid.

[1] [https://npmjs.org/package/stepup](https://npmjs.org/package/stepup)

[2] [https://npmjs.org/package/trycatch](https://npmjs.org/package/trycatch)

[3]
[https://github.com/joyent/node/issues/5114](https://github.com/joyent/node/issues/5114)

[4]
[https://github.com/joyent/node/issues/5149](https://github.com/joyent/node/issues/5149)

~~~
spion
> __Promises attempt to resolve this by coercing all exceptions to error
> passing, but not all exceptions are safe to be passed. Specifically, any
> exception originating from core that is not an invalid argument exception
> (think 4XX vs 5XX class errors) cannot safely be caught / coerced /
> continued on. Also, not all errors passed to callbacks are even
> theoretically passable as the unofficial policy of core is that core
> callback errors are only distinguishable from exceptions in that they occur
> after the origin tick. In other words, they are equally crash-worthy and
> equally non-crash-worthy. __

This is not true. With promises, the wrapper of core functionality is left up
to you. You can either write a crashy wrapper, an uncrashy wrapper, or a
wrapper which picks whether to crash or not depending on the kind of error. I
wrote more about this here [1]

Most default wrappers provided by promise libraries catch all synchronous
errors but don't do anything with asynchronous errors. So far, this seems to
be a fine default -- most unrecoverable, state-corrupting thrown errors in
node core are asynchronous, and all the invalid argument errors are
synchronous. But even if that weren't the case, it would be a simple matter to
write a more specific wrapper.

[1]:
[https://github.com/petkaantonov/bluebird/issues/51#issuecomm...](https://github.com/petkaantonov/bluebird/issues/51#issuecomment-31125928)

~~~
eldude
You're talking about the core/userland boundary beyond the origin tick, and
we're both right. Promises coerce caught exceptions occurring on the origin
tick to errors (bad), while allowing non-caught async errors to be handled in
a custom manner as you point out.

The comments following your linked comment address this nuance. I agree with
Raynos that the core of the issue is as I point out here, 2 divergent
incompatible error handling mechanisms, with both of them fundamentally
broken:

* error passing fails because we don't have CPS due to lack of Proper Tail Calls

* throwing fails because we lack async try/catch or at least with async generators we lack a performant try/catch or with bluebird-like optimizations (hacks) we lack typed catch and Error.create

The latter is far closer to a consistent error handling pattern than the
former, which promises implement (poorly and verbosely IMO). Additionally,
async generators also nicely address the issue of slicing your userland stack
away from core, so you get a two for one.

~~~
spion
Looks to me like you didn't even read the crashy wrapper. With it, its
possible to control what gets caught and what doesn't.

Although, I would really like to see a library that is written so badly that
even catching exceptions thrown in the same tick would make it unusable.

The default is fine, and exceptional cases can also be covered on demand. (The
presented `using` and `acquire` functions might even let you clean up after
some "non-recoverable" errors!)

~~~
eldude
FWICT, I fully understand the example and addressed your points directly. You
can wrap callbacks to catch everything or always crash, aka

    
    
         while allowing non-caught async errors to be handled in a custom manner as you point out.
    

And the referenced github issue furthers my point,

    
    
       Promises coerce caught exceptions occurring on the origin tick to errors (bad)
    

first via the OP[1]

    
    
        Is it possible to turn try { ... } catch off in the bluebird promise implementation?
    

and then via Dominic Denicola's characteristic snark[2]

    
    
        class PromiseThatMissesThePointOfPromises {
        ...
    

Please point out anything I am misunderstanding.

[1]
[https://github.com/petkaantonov/bluebird/issues/51](https://github.com/petkaantonov/bluebird/issues/51)

[2]
[https://github.com/petkaantonov/bluebird/issues/51#issuecomm...](https://github.com/petkaantonov/bluebird/issues/51#issuecomment-31076299)

~~~
spion
Then I don't see how you can claim that the fact that "Promises coerce caught
exceptions occurring on the origin tick to errors" is bad. This coerces all
the invalid argument exceptions into errors. Is that bad? Or are there other
errors which should not be caught but are thrown in this manner, and if so,
can you provide (links to) examples which ones and why?

These should be quite rare and are addressable by wrapping that badly behaving
function with a crashy wrapper that throws asynchronously...

~~~
eldude
I addressed your point and you are acknowledging it and I can see you
understand it from you various other responses. Promises catch "synchronous"
exceptions, however unlikely. You can optionally rethrow them / manually
crash, which you would need to. Your response is also the most common
response, "don't worry about them and deal with them when they come up."

I prefer to avoid this mentality and it motivates the difference in our
opinions.

~~~
spion
No, I'm just saying that undocumented inconsistency cannot be addressed in any
other way other than dealing with it on a case by case basis, as it comes up.

------
Patrick_Devine
I'm not a js guy, but this concept sounds a lot like Twisted's (python)
concept of "deferreds". The idea is that you "defer" caring about some
asynchronous process, and then attach "callbacks" and "errbacks" to the
"deferred". You essentially create a chain of callbacks or errbacks which gets
called depending on if a callback is fired or an exception is raised. Overall
it's a pretty great concept, but debugging can be a real pain.

~~~
domenicd
Yup, from what I recall of history both Twisted Deferreds and JavaScript
promises have their heritage in E's promises.

------
SimHacker
And what's wrong with reinventing basic programming-language design a million
different ways? Are you complaining that it's too much like Lisp?

------
hatchoo
Aren't they made to be broken?

(Sorry, couldn't help it)

------
OhHeyItsE
JavaScript promises. Got it.

Wait - crashing processes? Denial of service? Wha? Why would JavaScript?...

Oh, this is about Node. Goodbye.

