

Twitter Heron: Stream Processing at Scale - mekishizufu
http://blog.acolyer.org/2015/06/15/twitter-heron-stream-processing-at-scale

======
pixelmonkey
"In the evaluation in section 7 of the paper, Heron beats Storm in every
single metric."

In every metric except perhaps the most important one for actual users:

Storm is open source, Heron isn't.

Also, the @TwitterOSS account indicated that there are "no plans to open
source in the short-term":

[https://twitter.com/TwitterOSS/status/605460461761396736](https://twitter.com/TwitterOSS/status/605460461761396736)

~~~
frankmcsherry
The paper contains the motivation: "For our needs at Twitter, we needed a
stream processing platform that was open-source, high-performance, scalable,
and was compatible with the current Storm API."

I guess 3/4 isn't bad.

------
djb_hackernews
> All the Stream Managers in a topology connect to each other to form an
> O(k^2) connection network

shouldn't that be !(k-1)? Or am I missing something about the architecture?

~~~
escaped_hn
Every stream manager is connected to every other stream manager. k managers =
k^2 connections.

~~~
djb_hackernews
Yea I must be missing something here.

If you have k=2 stream managers you have 1 connection, not 4 . Even if the
connections weren't bidirectional you'd have 2 connections.

~~~
bmm6o
Parent is talking big-O and so is ignoring that it isn't exactly k^2.
Depending on how you count (do 2 nodes require 2 connections or 1 bi-
directional connection) it's k(k-1).

------
dang
Is the paper online? If someone can produce a standard URL for it, we'll
change the link in this post.

~~~
rckclmbr
[http://dl.acm.org/citation.cfm?id=2742788](http://dl.acm.org/citation.cfm?id=2742788)

~~~
dang
Sorry, I meant the pdf. Is there a url for it that will work for everybody?
The one I get when I click "pdf" on that ACM page seems unlikely.

~~~
frankmcsherry
SIGMOD 2015 proceedings are open access, meaning the conference has arranged
with ACM to provide the content to pretty much everyone.

That being said, the submission is to someone's (Adrian's) interpretation of
the work, and is arguably more useful than just linking at the pdf itself.

------
wheaties
Now I get it. It's not just "I didn't write this, therefore shit, so I need to
rewrite it from scratch." This was a clear explanation in a bite sized chunk
which illustrated the reasons for Heron. Thanks.

