

Experimenting with Node.js - jkreeftmeijer
http://jeffkreeftmeijer.com/2010/experimenting-with-node-js/

======
jamwt
While I think it's great that more and more developers are being exposed to
building network applications using async I/O (which I guess is "evented" now)
via Node.js, I think it's worthwhile to point out that the state of the art
has moved well beyond these kind of callback frameworks. The reason is simple:
it sucks programming in callback patterns on serious, large projects. You end
up with lots of routines that are 6 or 7 callback chained together--and don't
forget to attach error callbacks as well at each point.

In the Python world, for example, eventlet, gevent, and diesel (disclosure: my
project) all use coroutines to achieve very high performance asynchronous I/O
that's still written in a "synchronous" style. And before that, CCP games has
been using stackless python to do coroutine-based networking for the servers
behind the very popular MMORPG eve online
(<http://www.tentonhammer.com/node/10044>).

Even slicker, in languages like Erlang and Haskell (via forkIO + the
select/epoll I/O manager), support for asynchronous networking using
"blocking-style" code is a fundamental language/compiler/vm feature. When you
write _any_ code in these languages, you're transparently taking advantage of
the same scaling characteristics node.js and its ilk provide.

So, I'd humbly suggest to anyone getting serious about doing this kind of
programming that they investigate some of these systems (which are usually
built by weary callback-style async I/O veterans) and leapfrog the rest of
their peers. Consider Node.js a gateway drug.

Linkage:

<http://eventlet.net/> <http://github.org/jamwt/diesel>
[http://en.wikipedia.org/wiki/Asynchronous_I/O#Light-
weight_p...](http://en.wikipedia.org/wiki/Asynchronous_I/O#Light-
weight_processes_or_threads) [http://www.galois.com/~dons/slides/a-scalable-
io-manager-for...](http://www.galois.com/~dons/slides/a-scalable-io-manager-
for-ghc.pdf)

~~~
IsaacSchlueter
I've heard this before. "Oh, callbacks are fine for little things like this
hello world demo, but for Big Programs, you need (threads/coroutines/etc.)
because they're Serious Business."

That might be true. Maybe for Big Programs, something else is better. But I
think the problem is that setting out to build a Big Program for Serious
Business is Doing It Wrong.

The best frameworks (or, at least, my _favorite_ frameworks) are collections
of small tools with consistent interfaces and interchangeable parts.

If you can manage callbacks for a 100-200 LOC program, then you can build a
Big Program out of such modules with a little bit of forethought. Instead of
setting out to write a Big Program, why not try to figure out how to express
your Big Problem in terms of a bunch of Little Problems, and then come up with
a way to have Little Programs assemble.

Then, assemble the Little Programs to solve the Little Problems. There is no
solveable Big Problem which cannot be broken down into some finite set of
Little Problems.

Callbacks make it natural to solve problems in this manner.

I actively develop a several thousand line NodeJS project. It's quite nice,
actually.

~~~
jamwt
I don't know how to completely address your post b/c it responds to assertions
I didn't make. I said nothing about "Serious Business"--and bundling threads
and coroutines together is like bundling horseshoes and bicycles. I made no
defense of threads.

Callback style, within an I/O handler, is (almost always) a concession to the
event loop. When I'm writing a program, I write:

    
    
      routine():
        result = do_one_thing()
        do_next_thing(result)
        do_whatever()
    

With callback based I/O, you need the routine to _end_ so the "reactor" up the
call stack can get control again to do I/O on your behalf:

    
    
      routine_1():
        return make_some_io_request(callback=routine_2)
    
      routine_2():
        return make_another_io_request(callback=routine_3)
    
      routine_3():
        return send_response_to_original_socket()
    

Now, if you were desperate, you could just call over to the reactor like this:

    
    
      routine():
        resp = do_io_for_me()
        next_resp = do_more_io_for_me(resp)
        return transform(next_resp)
    

.. but, there are (at least) two problems with this:

1\. You're making the stack deeper every time you "call over" to the
reactor... you will stack overflow eventually

2\. In most languages, the reactor doesn't have a way to "call back into" your
stack frame and resume it.

Coroutines are a way to "call over" to the reactor, have your stack state
frozen (in heap space, without going deeper in the call stack), and then
"unfreeze it" later on, when the reactor has completed the IO.

The greenlet package provides this for Python; Haskell and Erlang have native
support for this. And, this is ultimately how you want to program. A(); B();
C(). Sure, there are exceptions for UI driven callbacks, and callback patterns
aren't all bad, but typically in network IO cases, the callbacks are
explicitly a concession to the I/O loop's need to be scheduled, made necessary
by a missing language or vm feature.

And: cleaner code benefits 200 LOC apps as much as it does 200k LOC apps. A
good design is a good design at any scale.

~~~
rictic
Could you be a bit clearer about what's going on in your third example?

~~~
ralph
I think he's saying that the main event loop has called a callback, only to
have that callback either call the main event loop, or something similar that
processes events until a certain condition, and that in turn may cause other
callbacks to be called. Stack depth has increased. And it tends to lead to
bugs because sometimes unconsidered things happen in the midst of the first
callback's processing.

------
reeses
Why is it that within ten seconds, I felt compelled to "hump" other arrows
with my own?

It was also interested to see if I could get other people to form lines and
other constructs, without any communication other than my own cursor.

~~~
WilliamLP
I'm with you that I found the social implications of this many times more
interesting than the technical ones!

------
weixiyen
I made this RPG node demo a while back: <http://sleeperbot.com>

I'm not using any server monitoring so it can crash at any moment.

Would love to get more than 15 users on here moving around and see what
happens.

code: <http://github.com/weixiyen/avatar>

~~~
JakeSc
I would highly recommend sanitizing chat input. As a quick demo, type
'<script>alert("a")</script>' or something.

Very cool concept though.

~~~
weixiyen
thanks, i put that in.

the blinky body toggle was pretty cool to see though :)

------
Kilimanjaro
Make it funnier, show us a maze and people trying to solve it in real time.
Replace the mouse pointer with a colored token and constrain its movement to
the walls of the maze.

~~~
axod
This was a startup idea I had when I started Mibbit (I decided on balance
webchat had more potential than multiplayer web games).

The idea was 'mouse games'. I had a similar setup where you could see
everyones mouse cursor. I planned a whole load of multiplayer games where you
could chase each other, complete puzzles, click on other people to kill them,
draw circles round other players to trap them, etc etc. You could have
messages show up like "First to draw a square gets a point". "Move to the left
get a point". "Move as far away from everyone else as you can. Most isolated
player gets a point". You could have objects being thrown at cursors, and you
have to dodge them. Tons of potential to be a fun way for people to waste time
:)

I'm sure it would have been great fun to make :)

~~~
icey
It still sounds like a pretty cool idea to me - I'd certainly check it out if
something like that existed.

~~~
axod
But would you also:

    
    
      * Come back regularly to play it?
      * Pay to play it?!! :/
      * Click on ads? (If the creator were evil, they'd confuse
        the users into clicking on ads, making them think it's
        part of the game).
    

Still a fun thing to make though!

~~~
icey
As with everything, it depends on the implementation ;)

I think it's probably tougher to monetize a web game, but flash game creators
seem to be doing moderately well with advertisements that run before you can
play the game.

There's an interesting hook in what you're talking about: I could IM a friend
and say "hey jump on here, let's play together". It's much more social than
most web games out there which is intriguing.

In general most web pages feel like you're there by yourself. The model of
seeing many other people there at the same time is novel enough right now that
it should be a little easier to get people to tell each other about it.

------
CitizenKane
Extremely cool demo. It would be interesting to see this applied to analytics,
so you could see exactly what people are doing on your site in real time.
Although that would also be a bit creepy.

~~~
riklomas
There's already something called Hummingbird that does that:
<http://mnutt.github.com/hummingbird/>

~~~
CitizenKane
Hummingbird shows traffic in real time. It doesn't show mouse movement or user
activity in real time.

------
JakeSc
I am thoroughly impressed, and excited for the prospect of unimaginably
awesome web applications. I do have a few thoughts:

\- How easy would this be to implement in another event-driven server, such as
Tornado?

\- I have an identifiable "mouse ID" in this app. Is this secure? Surely the
security aspects of web sockets have been key in their design, but I am left
with a bit of uneasiness with my browser opening a TCP connection to another
server. I believe WebSockets will pave the way for a whole new class of web-
based attacks, and possibly even highly mobile worms.

------
Joeboy
Please could you move your cursors out of the way? I'm trying to read the
text.

Edit: This lame comment is the highest rated here? You disappoint me.

~~~
jkreeftmeijer
There's a "disable" button in the yellow box. That'll make them go away
completely. :)

------
MisterWebz
I'm wondering if Javascript will start being heavily used server-side? Would
it be wise to start learning JS and how to use Node.js or is it absolutely not
suitable for production use?

~~~
jorangreef
Regardless of the popularity of Javascript on the server-side, if you're
writing a web-based application, then your architecture will benefit from:

1\. Shared code between client and server.

2\. Specialization in one language.

V8 and Node are at least a very good combination for server-side Javascript.

Re: "is it absolutely not suitable for production use?"

Node in alpha state is already more production ready than many "production
ready" tools.

------
kylemathews
The cursors looked disturbingly like flies. Pretty insane demo.

------
drats
amusing: <http://i.imgur.com/qHWYm.jpg>

------
DaemonXI
<http://i.imgur.com/OCaBf.jpg>

I feel OK about doing this.

~~~
einaros
I feel conflicted about doing this:
[http://www.youtube.com/watch?v=jULOA7mSOac&fmt=22](http://www.youtube.com/watch?v=jULOA7mSOac&fmt=22)

------
ppplll
It would be interesting to hear from the author how the server is handling
HackerNews traffic

~~~
jkreeftmeijer
Hi, author here. While building this thing, the Node server had the tendency
to drop out. I'm monitoring it with God (<http://god.rubyforge.org/>) now, so
it restarts whenever anything goes wrong.

I'm genuinely impressed by how it's holding out now. :)

~~~
jokull
Could you elaborate on this at all? What kind of traffic are you getting (max
concurrent requests handled perhaps)? Any idea why Node dropped out?

------
aslakhellesoy
Web sockets are great, but almost all of the demos I see use a separate port
(i.e. not port 80), making the app useless in practice.

Come on guys - it's _Web_ sockets, not intranet sockets.

~~~
lftl
Could you elaborate on why a nonstandard port makes these apps useless?

~~~
user24
corporate firewalls blocking nonstandard ports, you running other apps on
those ports, your firewall throwing a wobbly, general nonstandardness.

if 80 is impractical, many people suggest running off 443 as that (the https
port) is unlikely to be filtered.

~~~
aslakhellesoy
It's not hard to serve both regular http requests and web socket requests on
the port 80 on the same host. Just use a special context path for web socket
requests, for example /websocket/*

Some of the node.js websocket impls let you do this easily while others don't.
Don't use the ones that require a separate port. Nobody behind a corporate
firewall will be able to use your app if you do.

~~~
lenary
I think it is possible to have a ws server listening on the same port (and
even path) as an http server, hence it shouldn't really matter.

And if you really care, you could make a http forwarding proxy with node to
forward normal http requests through to your app if it's written in something
else, but deal with websockets itself.

------
nhnifong
It a strange mix of disappointment and excitement to find out someone has just
released a better finished version of what you were working on. Oh well, I
guess I'll finish it anyways.

~~~
jkreeftmeijer
Please let me know when you finish it. I would love to see your code. :)

~~~
nhnifong
It has mutated quite a lot since I started... I'm taking the positions of
everyone's mouse and feeding them to Fortune's sweep line algorithm to make
Voronoi diagrams. I've made it look like you are a cell squeezing through
layers of tissue. <http://uncc.ath.cx/applets/Voronoi_Standalone/>

------
igrigorik
Great stuff.

Just for sake of completeness, you can basically copy-paste that code into
Ruby as well via em-websocket: <http://github.com/igrigorik/em-websocket>

Also, to avoid some of the callback muck that a few have brought up:
<http://github.com/igrigorik/em-synchrony>

(same idea, closures + coroutines to abstract the callbacks).

------
olalonde
Is the server down? I cannot see any arrow even when I open multiple windows
(Chrome 5).

~~~
maushu
Check your firewall, the websocket doesn't use the port 80.

------
olalonde
For some reason, the background picture is distorted when remote mouses hover
it (Chrome 5). Anyone has an explanation?

