

Node.js stream handbook - hoggle
https://github.com/substack/stream-handbook

======
hoggle
Also by the magnificent substack, learning by doing done right: the node.js
stream adventure!

Install via:

    
    
       npm install -g stream-adventure
    

Now simply go forth and start your adventure by typing:

    
    
       stream-adventure
    

Have fun and don't forget to thank the man:

[https://github.com/substack/stream-
adventure](https://github.com/substack/stream-adventure)

[https://twitter.com/substack](https://twitter.com/substack)

------
shortstuffsushi
Not hoping to start any sort of flame war, but could someone try to explain
this difference between streams and promises? The conceptual difference
doesn't seem completely clear to me.

~~~
todd3834
Promises hep deal with asynchronous functions by providing a convenient way to
pass success/fail methods in a way some people prefer more than classic
callbacks.

Streams are a totally different use case. Streams are used when you want to
process data one part at a time without having to have all of the data in
memory at once.

*edit: typo

~~~
muraiki
Promises are actually about trust, not syntax. With a callback you have to
trust that the function that will invoke your callback (which might not be
something you wrote) will only ever call it once, that it will pass through
errors, that it won't call both your success and failure callbacks, etc.

With a promise (or rather, following the promise specification) you instead
get back an object that you choose how to handle. That object is either
pending or else an immutable success or failure result. Either success or
failure will be called (not both), and whichever result is called can only be
called once. As such, promises allow you to avoid inversion of control and to
safely interact with potentially untrusted code.

This great series of articles helped me understand this:
[http://blog.getify.com/promises-part-2/](http://blog.getify.com/promises-
part-2/)

------
dpweb
This is a great write-up on streams. For curiosity sake, I'd be interested to
know the overhead of

    
    
        var stream = fs.createReadStream(__dirname + '/data.txt');
        stream.pipe(res);
    

vs. fs.readFile('./data.txt) and res().

and if streams become a better and better option as response sizes get larger
and larger. The streams syntax is nicer looking in any case.

~~~
sehrope
> ... and if streams become a better and better option as response sizes get
> larger and larger. The streams syntax is nicer looking in any case.

fs.readFile(...) reads the entire contents of the file into memory. For small
files that may be acceptable but if you're dealing with anything that could be
large it's not going to be very pleasant or even work properly.

I just tried it out on my laptop for a 100MB and a 1GB file. For a 100MB file
using streams is about 25% slower. However the sync method failed for the 1GB
file:

    
    
        Style            100MB       1GB
        =====            =====       ===
        fs.readFileSync  .176s    <failed>
        streams/pipe     .234s      1.25s

~~~
hoggle
By adapting the highWaterMark option one could probably tweak it even further.

Default buffer size seems to be 64k:
[https://github.com/joyent/node/blob/912b5e05811fd24f09f9d652...](https://github.com/joyent/node/blob/912b5e05811fd24f09f9d65200a1561a4482f166/lib/fs.js#L1468)

Stream buffering:
[http://www.nodejs.org/api/stream.html#stream_buffering](http://www.nodejs.org/api/stream.html#stream_buffering)

------
yudong
Will it work with LIVE video streaming?

For example, can you use ffmpeg to pull live video via rtsp, re-encode and
pipe to nodejs for it to stream down to browser and to be consumed by html -
with or without a plugin?

Any working code sample?

~~~
gabipurcaru
Yup, it does. I do have a quick working example, with a plain multipart form
:) You can check it out here:
[https://github.com/gabipurcaru/peggy](https://github.com/gabipurcaru/peggy)

While this is not exactly what you are talking about, I think it demonstrates
that node.js streams can do that too.

------
fibo
Can't believe it was not in HN yet

~~~
alphonse23
it was, it's been posted several times -- at least I know for sure in
comments. I was about to ask, "why are so many links getting re-posted in
hackernews?"

