
Heron Donated to Apache Software Foundation - tylertreat
https://blog.twitter.com/engineering/en_us/topics/open-source/2018/heron-donated-to-apache-software-foundation.html
======
mabbo
> Today, we are proud to donate Heron to the Apache Incubator where the
> community will continue to grow and thrive under the guidance of the Apache
> Software Foundation

Why do I keep reading this as "Today, we are proud to donate Mittens the cat
to the local animal shelter where the kitty will continue to grow and thrive
under the guidance of the SPCA."

~~~
rectang
When I was doing a lot of ASF administration (Board, VP Incubator, VP Legal),
I was always keenly aware that the core personnel on the projects coming to
the ASF were sensationally brilliant engineers and charismatic thinkers. The
idea of "guiding" such people like cats... is just way weird.

------
politician
Given the perception (fair or not) that the ASF is the place that large
projects go to die, I'd like to know why this donation is a good fit for the
ASF and what resources are in place to help it flourish there.

~~~
tveita
If potential users are running on Mesos, streaming data from Kafka and writing
results to HDFS, they're presumably not that worried about relying on Apache
Foundation managed software.

Not to say that this project won't die - most projects do, but I think the
"where projects go do die" stereotype is unhelpful. ASF is probably the best
chance a project has to move beyond the control of a single company.

------
cozos
My company is considering "upgrading" our Storm topologies to Heron. Given
that Storm is Heron's predecessor, and Storm itself is maintained by the ASF,
I wonder what this donation means for Heron.

p.s. does anybody have experience with Dhalion, Heron's autoscaling thing (is
it a module? or a plugin, idk). All I could find on it was a research paper
and a barren github repository; how do you use this thing?

~~~
avflor
Great! Thanks for the questions. The Dhalion repo
([https://github.com/Microsoft/Dhalion](https://github.com/Microsoft/Dhalion))
contains only the general Dhalion APIs. These are not Heron specific, that's
why they are not part of Heron. In Heron, the healthmgr module essentially
integrates these Dhalion APIs with Heron. Currently, Heron contains two
Dhalion policies. The first one automatically restarts instances that exhibit
backpressure. The second one scales up or restarts instances depending on some
topology characteristics. Please join slack and we can show you how to
use/modify the appropriate Heron yaml files so that you can use Dhalion.
Dhalion does not require a separate installation -- it is already integrated
with Heron and just needs to be configured depending on your use case. We are
currently working on more aggressive autoscaling policies which should be part
of the code base soon. Please let us know if you have other questions.

~~~
cozos
Thanks for the reply!

For us, the ability to self-tune is one of Heron's most attractive features.
Tuning components/executors in Storm was a big pain point. Have you guys ever
tried using the Dhalion autoscaling combined with hardware autoscaling from a
cloud provider (i.e. Azure, AWS EC2)?

~~~
avflor
No, we have not used auto-scaling with automatic hardware provisioning. We
assume that the hardware is there and we just request more containers (e.g,
YARN containers). Theoretically, it should be possible to do that if the
autoscaling policy is extended to request for additional hardware before
changing the parallelism of a stage. But this functionality is not currently
there.

------
qwerty456127
RIP Heron (;,;)

