

Wait.for: Sequential programming for Node.js (beta) - lucio
https://github.com/luciotato/waitfor
Simple, straightforward abstraction over Fibers.<p>By using wait.for, you can call any nodejs standard async function in sequential&#x2F;Sync mode, waiting for result data, without blocking node&#x27;s event loop (thanks to fibers)<p>I&#x27;m looking for real-world node apps, to test wait.for
======
chrisohara
Why not use harmony generators?

Generators are provided by V8, don't require an external fibers extension
(i.e. hack), and will be available in the next node stable via `node
--harmony-generators`.

Here's an example for those interested:
[https://github.com/visionmedia/co](https://github.com/visionmedia/co)

~~~
lucio
node-fibers is under the hood, and works on the actual node stable. I hope ES6
generators -when stable- will provide similar tools, so wait.for will be based
on ES6 generators.

------
FooBarWidget
> What if... Fibers and WaitFor were part of node core?

> then you can deprecate almost half the functions at:
> [http://nodejs.org/api/fs.html](http://nodejs.org/api/fs.html) (clue: the
> Sync versions)

Not quite. I use the Sync versions not to avoid callbacks, but to _ensure that
nothing else happens while I 'm doing the operation_. For example, take a look
at the Database.reload function from this project (a NoSQL database written in
Node):
[https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...](https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L103).
Reloading the database contents involves traversing multiple directories and
reading multiple files. While reloading the database contents, I do not want
the database to be modified. With asynchrony, the amount of different states
that I have to be aware of rises so quickly that it's much, MUCH easier to use
sync calls and tell the user "reloading is not concurrent, deal with it; at
least it's correct". Wait.for does not solve this problem.

It seems people believe that you can't run into concurrency problems or race
conditions because Node is single threaded. This is false. I even had to write
my own locks
([https://github.com/phusion/zangetsu/blob/master/lib/zangetsu...](https://github.com/phusion/zangetsu/blob/master/lib/zangetsu/database.js#L509))
because Node doesn't provide one.

------
casual_slacker
Cool idea. You should mention upfront that you're using Fibers under the hood,
since it has important performance implications. Also, your Wait.for function
only takes functions right now; but if you check for a .then function you can
accept promises too.

~~~
lucio
Thanks. I've put a link to node-fibers, 2nd line of readme. ¿are Fiber's
performance problems solvable? ¿what if fibers were integrated in node core?

~~~
AdrianRossouw
fibers will never be integrated into node core.

for what it's worth, I also refuse to use or depend on any code that makes use
of node-fibers. I know i'm far from the only person who feels that way too.

------
olegp
If all you want to do is make existing async methods synchronous, it's easier
to modify the function prototoype like this: [https://github.com/olegp/mongo-
sync/blob/master/lib/mongo-sy...](https://github.com/olegp/mongo-
sync/blob/master/lib/mongo-sync.js#L12) or just use Futures that come with
fibers.

However, in practice you want a higher level of abstraction, since multiple
async calls typically end up mapping to one sync call. This applies to streams
in particular where you need to wait until they become readable.

I've written Common Node ([https://github.com/olegp/common-
node](https://github.com/olegp/common-node)) to do just this and made it
CommonJS and RingoJS compatible, which means the code written for Common Node
runs on both V8 and the JVM. We've been using it in production at
[https://starthq.com](https://starthq.com) for a while with great success. Due
to some optimizations we put in place, Common Node actually performs faster
than Node in some benchmarks.

------
scriby
[https://github.com/scriby/asyncblock](https://github.com/scriby/asyncblock)

I'll just leave this here. It's been my pet project for the last couple years,
and is a fully featured version of this concept.

There's also [https://github.com/0ctave/node-
sync](https://github.com/0ctave/node-sync), which is the same concept.

------
zongitsrinzler
Correct me if I'm wrong but isn't a big part of async programming that it
forces you to think about error handling. Working as a Java developer I have
seen very lousy error handling mostly because try/catch lets you wrap
everything and kill errors easily. Most of the node code I have seen handles
errors much better thanks to it's async nature.

~~~
gruseom
If you do it well, you can get very precise error handling, because errors are
detected in the context where they first occur. However, to do it well you
have to put a try/catch around every asynchronous operation, which makes
nested callbacks even harder to read than they already are. Consequently
people write code that doesn't handle all possible errors, and when one falls
between the cracks, it gets caught only by the top-level default handler
(assuming you've declared one), at which point the context is lost.

The other problem that comes up is the difficulty of getting a backtrace
beyond the current callback. It can be hard to troubleshoot an error in a
callback when you can't look back at how the async call got started. In such
cases what you really want is a list of backtraces that capture all the
relevant history. I seem to recall that people have come up with solutions for
this, but can't remember what they are. Maybe someone else can comment?

~~~
tg3
Most Node.js I've seen doesn't make extensive use of Javascript's built-in
exception handling, and instead relies on the convention of the first
parameter of a callback being an Error object if one occurred (so no
try/catch, which doesn't catch async errors anyway).

The stack trace is a pain for those operations, and any error that occurs as
part of "normal" javascript operation (e.g. ReferenceError) has to be handled
like a normal exception, which makes the style inconsistent and therefore a
bit harder to manage.

I think that the latest or upcoming version of Node is presenting a solution
to this in the form of domains, but I'm not too familiar with it.

~~~
gruseom
The first-parameter convention only works for application errors and errors
from libraries that also follow the convention. Everything else, including the
JS runtime, throws exceptions whether we like it or not. So either you have to
make sure that your code never triggers any of those—which is a pretty tall
order—or put try/catches around everything, even if your own code doesn't
throw any exceptions. That's more than a bit harder to manage—if you do it
correctly, there are so many try-catch blocks that the rest of the code almost
gets drowned out by them. In my observation, most people deal with this by
sacrificing a certain amount of error safety in order to make the code more
tractable.

It seems like the kind of problem that is better dealt with by a compiler.

~~~
aaronyo
Promises help with this quite a lot. [http://promises-
aplus.github.io/promises-spec/](http://promises-aplus.github.io/promises-
spec/) What 3.2.6.2 means is that any thrown exception is caught and turned
into a rejected promise which will bubble up (across async boundaries) until
it gets to a rejection handler.

I haven't had too much experience with Domains, but I believe they are
generally used for courser grain error handling (because it's a bit verbose to
attach them all over the place). E.g., to catch most errors (it's no guarantee
what 3p libs do to your context) generated in fulfillment of a particular web
request.

~~~
gruseom
Can promises guarantee that condition with only library code? or do you still
have to have try/catch in the application code, and if so where?

Also, do any of the promises libraries solve the backtrace problem, and if so
how do they do it?

------
ilaksh
Please also take a look at ToffeeScript which is a brilliant and amazingly
relatively unknown solution. [https://github.com/jiangmiao/toffee-
script](https://github.com/jiangmiao/toffee-script)

~~~
SeanDav
Not dissing your comment, but I am not a believer in any "compile to" language
because ultimately when things go wrong you are forced to deal with the
underlying language in any case. Rather become an expert in the underlying
language and learn to deal with its idiosyncrasies.

~~~
ilaksh
I'm not suggesting that people shouldn't learn JavaScript. Anyway all
languages compile are abstractions of some lower level (usually multiple
levels).

------
tspike
Is this different from async.series? If so, in what way?

~~~
lucio
\- syntax

\- wait.for it's simpler

\- wait.for uses try/catch

I don't know async.series deeply, but, ¿how do you use results from the first
function on the second or third?

wait.for:

    
    
      try{
        data = wait.for(function1);
        result = wait.for(function2, data);    
      }
      catch(err){
        // error handling
      };
    

async.series:

    
    
      async.series([
        function1,
        function2
        ],
        // optional callback
        function(err, results){
          // results is now equal to ['one', 'two']
        });
    

_async_ has a lot of functionality. You can use _wait.for_ and _async_ at the
same time. They supply different functionality.

Example:

    
    
      results = wait.for(async.map,['file1','file2','file3'], fs.stat);

~~~
htilford
> ¿how do you use results from the first function on the second or third?

waterfall

    
    
      async.waterfall([
        function (cb) {
          . . .
          cb(stuff);
        },
        function (stuff, cb) {
          . . .

------
ibizaman
Wouldn't it be easier to just not use Node.js then ? :p

------
EGreg
I don't get it,

    
    
        console.log(wait.for(fs.readfile,'/etc/passwd'));
    

won't that execute console.log immediately, whereas fs.readfile will return
the data later? "I don't think it means what you think it means"
[http://www.youtube.com/watch?v=G2y8Sx4B2Sk](http://www.youtube.com/watch?v=G2y8Sx4B2Sk)

~~~
lucio
wait.for, waits for fs.readfile to callback, and returns "data" from
callback(err,data)

wait.for(fs.readfile,'/etc/passwd') == fs.readfileSync('/etc/passwd')

wait.for allows you to call any async function as if it were a sync function,
without blocking node's event loop. It works inside a fiber. Check the
examples.

~~~
unnuun
Right, but how the heck is this achieved? Could you show a line or two of
example pseudo code that demonstrates the core of this functionality?

~~~
tg3
It uses Fibers ([https://github.com/laverdet/node-
fibers](https://github.com/laverdet/node-fibers)). I don't know much about it,
but you have to wrap all your code in a call to `Fiber()`, which is written in
C++.

~~~
lucio
Fiber() is under the hood, and I hope ES6 generators -when stable- will
provide similar tools, so wait.for could be based on ES6 generators.

You don't have to wrap all your code in a call to `Fiber()`

full classic node-server sample:

    
    
        var http=require('http'), wait=require('wait.for');
    
        http.createServer(function(req, res) {
          wait.launchFiber(handleRequest,req, res); //run in a Fiber, keep node spinning
        }).listen(8080);
    
    
       //in a fiber
       function handleRequest(req, res) {
         try {
           res.writeHead(200, {'Content-Type': 'text/html'});
           res.end(
           "<h1>Demo page</h1>" 
           + "<p>"+ wait.for(fs.readFile,'/etc/somefile') +"</p>"
           + "<p>"+ wait.for(fs.readFile,'/etc/otherfile') +"</p>");
         }
         catch(err) {
          res.end('error '+e.message); 
         }
       }

~~~
tg3
in your example below, you do have to wrap it in a call to `Fiber()`, although
you've disguised it as `launchFiber`, which you pass `handleRequest` to (I
would call that wrapping `handleRequest`.

That is probably a worthwhile tradeoff, but the first time I read through it,
that mechanic wasn't obvious to me.

~~~
lucio
It's not 'disguised', _it 's abstracted_. _launchFiber_ does what it's name
implies: launchs a fiber. How it is achieved 'internally', it's irrelevant as
long the abstraction is not leaky.

Today _launchFiber_ is calling the fn inside a Fiber(), tomorrow could be
launching the fiber by means of a generator.

The idea was to create a _very simple_ abstraction to avoid callback hell,
adding low or no complexity, and keeping the flexibility to use all the
existing async fns with a callback or just "wait" for the results.

