
Node Streams: How do they work? - bpierre
http://maxogden.com/node-streams
======
malone
Streams are nice but in my experience there aren't many modules that let you
use them. Most networking protocol modules will only let you specify a
hostname and port to connect to, and not pass a stream to use. That's fine for
the common case, but maybe I want to do something fun like send data encrypted
with ssl, which is encoded using base 64, then stuffed in a dns query, then
tunnelled over socks, via a unix pipe (and receive the reply via a completely
different stack). It wouldn't be hard to chain streams together to achieve
that, but then I'd have to rewrite modules to actually use it.

Even nodes own http module only does standard tcp. If it supported arbitrary
streams then it'd be much more flexible, and we wouldn't need to do things
like use a separate module to support https.

The same with the server part of these modules. They normally only let you
specify a tcp port to listen to. Ideally there should be some sort of "stream
server" interface that automatically sets up the stream chain and emits a
connection event. The module could then listen for that event and use the
corresponding stream for transferring the data. That way the module would be
completely independent of the underlying protocol.

~~~
grncdr
LuaSocket's ltn12 module might interest you:

<http://w3.impa.br/~diego/software/luasocket/ltn12.html>

------
tantalor
Reminds me much of channels in Go:
<http://golang.org/doc/effective_go.html#channels>

Are Node streams pure, generic JavaScript? Do they rely much on Node internals
or outside libs?

I ask because I experimented with "porting" Go channels to generic JavaScript
in <http://www.johntantalo.com/blog/go-flavored-javascript/>

~~~
maxogden
they are pure js, though streams in node emit Buffer objects which can be
mostly emulated using html5 Int32Arrays

~~~
tantalor
Had to look that up,

> A Buffer is similar to an array of integers but corresponds to a raw memory
> allocation outside the V8 heap.
> <http://nodejs.org/docs/v0.3.1/api/buffers.html>

Are streams limited to binary buffers, or may they also emit JavaScript
objects or primitives? Binary may be great for low-level I/O streams, but I'd
expect more nuanced events for high-level streams.

~~~
maxogden
You can emit anything but it just so happens that 99% of the streams used in
node core are Buffers for performance reasons (Buffers are allocated outside
of V8's heap). An example of emitting objects is in here
[https://github.com/maxogden/dominode/blob/master/example.htm...](https://github.com/maxogden/dominode/blob/master/example.html)

------
0x5a177
Using Node has helped me think more in terms of streams because the APIs
afford such thinking. PHP has streams but I never thought to use them, and I
haven't really seen it done.

Node Streams really help performance in my application where I am proxying API
calls from the browser to the Dropbox API. It's much faster to stream the
response from Dropbox through to the client rather than request, wait for
entire response, then send.

------
wizard_2
I've had trouble wrapping my head around how to use one stream to filter data.
I'd like to be able to write to a stream and read from it to get the filtered
data but the api only seems to lend itself to using two streams (think stdin
and stdout).

------
kev009
magnets.

