

Introducing NowJS or "How to make a chat server in 12 lines of code" - sthatipamala
http://nowjs.com

======
substack
Rival node.js RPC project (dnode) author chiming in here.

This is pretty neat and I like the emphasis on getting stuff going in as few
lines as possible but I'm not convinced that return values and implicit result
callbacks are the way to go for asynchronous requests, which is how I
understand this project to work. Often when you want to make a remote call
you'll be doing some I/O on the server side, which in node is asynchronous so
you can't return right away. Passing along a callback from the client side
works much better when the I/O actions have some values to call the client
back with.

~~~
ericz
Hey substack,

The implicit callback is definitely my least favorite part in building NowJS.
You can still pass in an additional callback to do any asynchronous I/O. We're
trying to find ways to make this experience better for the next version

~~~
substack
The solution I settled on for dnode is that a deep traversal finds all the
functions in your argument list, pulls them out, and wraps them. This part of
the protocol is a reusable module if you want to go with it:

<https://github.com/substack/dnode-protocol>

It's written for continuation-passing style but it shouldn't be too hard to
retrofit it for other call styles if you want to preserve explicit returns for
instance.

~~~
ericz
Awesome. I'm looking into this for next release

~~~
beej71
You guys have the hacker nature. I dig it.

------
glesperance
Blending this with browserify + backbone + redis will be really easy to do. I
think it will definitely simplify the dev of our app ; allowing us to share
even more code between the client and server so that we can maximize code
reuse.

~~~
substack
On that note, (browserify author here) I would love to see projects like nowjs
start to abuse the "browserify" field in the package.json for client
distribution like dnode v0.6 does (presently in staging until socket.io-node
v0.7 lands). With this field you can `require('pkgname')` in your browser-side
code just by specifying 'pkgname' in your require list, even though pkgname
might have both server and browser-side components.

~~~
sthatipamala
Sounds like it would be amazingly convenient. We'll definitely look into that
for next release.

------
dhruvbird
Great stuff!! So, do you open up 2 channels? One for the client to make calls
to the server (this could just be normal HTTP) and one for the server to make
client calls?

~~~
sthatipamala
Depends. We're built on top of Socket.IO (<http://socket.io/>) which uses a
websocket connection when possible. It falls back to assorted other transports
if websockets are not available.

~~~
substack
And in the next release of Socket.IO Guillermo has promised us multiple
channels, so projects like nowjs should get channels for free!

------
oomkiller
Wow, I've already thought of a bunch of cool ways I could use this. Although,
it does remind me of runat="server" for some reason. :)

~~~
sthatipamala
Yes, we've taken a little inspiration from Jaxer. But NowJS is much friendlier
since you have the power of the Node.js support.

~~~
Lennie
I think ASP was first to implement this.

------
catshirt
how does this compare to dnode?

~~~
substack
Nowjs seems to have a replication system baked in with harmony proxies so when
you update data structures these changes get pushed out to the clients. I
chose to have dnode just make copies but I do plan on writing a replication
middleware at some point with the .use() functionality that landed in dnode
v0.5 so I'm interested to see how well this replication approach turns out
with nowjs.

------
DenisM
All attempts to pretend that remote calls are local calls have failed thusfar.
Chances are you will regret going down this route.

~~~
sthatipamala
Could you present some examples? We'd be interested to learn from the past.

~~~
DenisM
Sure: dcom, corba, rpc, soap. The lesson I learned is that you can't abstract
away the network barrier; to the contrary - that barrier must be the central
point of the design of storage and communication components of the app.

~~~
substack
I suspect that previous attempts haven't been successful for a number of
reasons but especially because they tend to be synchronous. Node is a great
platform for asynchronous rpc because the asynchronous character is already
designed to make the programmer have to think about where the high-latency
barriers are. I ran into this problem with drb in ruby for a job queuing
system and had to write my own thread pool and polling system to compensate
for it. Handing a callback off to your remote method does seem to work
reasonably well at addressing these common rpc pitfalls.

~~~
jerf
"I suspect that previous attempts haven't been successful for a number of
reasons but especially because they tend to be synchronous."

No. Node.js has no special contribution to asynchronicity, which has been in
common use for something like 20 years now. Every UI-based program you have
ever used is asynchronous-event based, even if it has synchronous components,
to say nothing of the many many other Node.js-like libraries that have been
written over the past decade in other environments, many with more capable
languages that Javascript. None of these have been able to fix RPC, and many
of the biggest and most well-funded attempts like CORBA and COM have come from
smack in the middle of some of the largest piles of event-based code in the
world.

The reason trying to pretend a network transaction is a function call fails
has to do with the different semantics of a function call versus a network
transaction. Latency differences even when everything is working correctly
certainly are a factor, but are really less interesting than the problems of
the unreliability of the network and the fact you are crossing a semantic
boundary. Every network transaction can fail. Every network transaction might
succeed, but with unacceptable levels of latency. Every network transaction
might initially succeed but cutoff halfway through, or dribble its results in
one byte at a time, or send you a gigabyte unexpectedly, or any of a variety
of other failure cases you must at least be ready for, even if you can't
"handle" (because in some cases there is no "handling" them). Furthermore,
every network transaction incurs a serialization step, in which the semantics
of the local program must be re-enforced on the data, possibly with failures
thereto. For instance, you might get back JSON that specifies an integer
greater than 5 billion in size where your environment is still only 32-bit; no
local function call can do that. (Which isn't to say local function calls are
therefore perfect, it is just that their failures lie in other places. For
instance, your overlarge number got truncated somewhere else in your program,
but it wasn't a function call that did it, it was some actual math computation
somewhere. The point is that there is a different sort of semantic failure
that can occur with an RPC vs a local function call and this inevitably leaks
out of any abstraction you could try to wrap around the RPC.) And that was
simply one tiny example, not the totality of the issues you can encounter, the
vast majority of which are far more subtle than that.

RPC (where "procedure" is an old synonym for function) should by design _not_
deal with these issues if it's really going to be "RPC", because by definition
of RPC it really ought to look like a function call. Therefore it must handle
all of these issues implicitly, and since no one answer is adequate for all
cases, usually incorrectly. You can't seal over network issues anywhere near
well enough to make dealing with a network as easy as dealing with a function.
You must deal with latency issues, but again, these are ultimately the least
interesting aspect. You must be able to deal with serialization and semantic
issues; if your language forced you to deal with that on every function call
you'd never use it. You must have some way of dealing with the various network
failures and even throwing various appropriate exceptions only gets you a
subset of the actions you might actually want. You must, inevitably, allow
these things to poke through somewhere, at which point your are "configuring"
your RPC call, at which point it is really no longer a "procedure call" at
all, it's something else.

That it has historically imposed an additional point of synchronous behavior
in some cases is because languages up to this point have also been largely
synchronous, but that has nothing to do with RPC's far more fundamental
failures as a network communication metaphor. It is also the case that if
you've absorbed too much Node.js hype that you may underestimate the world's
understanding of the problem; take a moment to search for "asynchronous COM",
for instance. The first hit I get is an article from April of 2000. The
synchronous problem is easily and trivially solved by turning RPC calls into
futures objects instead, which has been done, and has not salvaged RPC because
it is not the core problem RPC has. And there are other solutions, too, which
also don't work.

Regrettably, calling things over a network _must_ be more complicated than a
local procedure call; all attempts to make it otherwise have indeed failed to
live up to their promises.

~~~
DenisM
I feel bad for you having type this well-reasoned argument only to have one
person appreciate it (beside me, didn't need any convincing to start with).

Couple of things I'd like to add are versioning problems (on the net old and
new versions of code talk to each other, which almost never happens in local
calls), and time-travel problem where an operation succeeds, only to have all
of its effects reversed because the remote server failed and was restored from
a day old backup.

Interesting thing about rpc is that it is so very sexy, and it is tempting you
to think that you will add error handling later. So it hides 99% of all
problems from you under a sexy appearance.

------
brosephius
n00b question here, but I installed it with npm and I'm trying to run the chat
sample, but node says it can't find ../lib/nowServerLib.js

I'm still trying to figure out node so pardon my ignorance, but am I supposed
to run this from a specific location? anyway, the project looks pretty cool,
looking forward to playing with it.

------
BobKabob
Very cool stuff!

I'd love to see an example as to how you'd integrate that into a Python/Django
project.

~~~
wampwamp
redis.publish(some_listener, payload)

------
moe
What about errors and exceptions? Will they propagate to the calling side?

~~~
ericz
In the typical node pattern, errors should be sent as part of the callback for
asynchronous IO. For exceptions, a "throw" statement will not be automatically
propagated to the other client, best practice is to catch the errors and call
a client side function in that case. Exception handling will be improved in
the future.

------
iag
Great stuff Darshan!

------
sinaiman
But will it blend?

