
Storm 2.0 - mnming
https://storm.apache.org/2019/05/30/storm200-released.html
======
avinium
Salient line:

 _While Storm 's Clojure implementation served it well for many years, it was
often cited as a barrier for entry to new contributors. Storm's codebase is
now more accessible to developers who don't want to learn Clojure in order to
contribute._

Very common story that I hear from a lot of shops that tried to go Clojure
first. Great language, but too little penetration amongst developers. Makes it
very difficult to effectively recruit employees/contributors.

At some point, you just accept that it's more important to get a new warm body
than to continue pursuing some idealized programming perfection.

~~~
dleavitt
This is salient too though:

 _The new Java-based implementation has improved performance significantly,
and made Storm 's internal APIs more maintainable and extensible._

It's not totally clear if this is _because_ it's been rewritten Java, but the
intrinsic qualities of the language do matter; there are real tradeoffs
between maintainability and dynamism/flexibility. I feel like this sometimes
gets shortchanged in these narratives.

~~~
bjoli
Idiomatic java is like idiomatic C: pretty fast. Most smaller languages
prioritize other things. Clojure has immutability and Haskell has purity.

I have noticed this with every "X produces code faster than C": you begin with
two programs that use a suboptimal algorithm, then you take your non-c
language and try to write C in it. The result is always awful and removes most
reasons not to use C in the first place.

This has somewhat changed with rust and in some sense C++,but for other
languages my point still stands. They have a nice idiomatic golden path that
is fast enough for most cases. Once you need performance badly enough you have
to treat you language as an assembler, and then you will always lose to
languages that actually are good at that.

I say this as a scheme/Haskell weenie. Writing really performance scheme and
Haskell code often means writing ugly code.

~~~
vbezhenar
Kotlin generally is not slower than Java. One exception is that it inserts a
lot of null checks. And there are features that will make it even faster than
Java, I'm talking about inline lambdas.

~~~
pvorb
The compiler is significantly slower than Java's, though.

~~~
vbezhenar
That's the sad truth for a lot of modern compiled languages (I think Go is an
exception). While I did not dig into their compiler internals, it just seems
an inevitable consequence from a powerful language (Go is pretty simple in
that regard). C++, Rust, Scala, Kotlin, Swift, they all have significantly
longer compilation times compared to their predecessors. Probably that's the
price we have to pay.

~~~
bjoli
Chez compiles a 80Kloc project of mine in less than a second, and apparently
that is still 2x slower than before they switched to a nanopass compiler.

There is no reason compiling things should not compile at similar speeds to
chez unless you are telling your compiler to ootimise everything.

C++/ruae are just exceptionally slow and it became the new bottom line
somehow. I know rust does a lot of housekeeping, but after using chez and
sbcl, 45s for a 1.5kloc project is killing me inside.

------
unlogic
To me, this makes total sense as the project moved to Apache. Obviously, much
more people will be able to consider contributing when it's in Java. Apache
goal is sustainability and long-term viability, and Java would work better for
that.

I also consider this a success story for Clojure. It gives Clojure another
usecase: a "production-ready prototype" language where the resulting
"prototype" can last for eight years and benefit thousands of developers until
it gets rewritten to something else when all the hard questions are answered,
and most experimentation/wandering is over.

~~~
usgroup
I think it’s noteworthy that whatever the benefits of clojure it didn’t
outweigh the adoption issue.

It’s quite a mature language so I’m not sure that bodes we’ll for its
prospects this late in the game.

~~~
yogthos
A Java shop inherited a Clojure project and rewrote it in their preferred
language. It's silly to read more than that into this.

~~~
usgroup
I’d agree if it didn’t keep happening .

~~~
sooheon
Rewrites between languages keep happening in all kinds of permutations. I
think pessimism or optimism from just headlines and announcements is more
about what you read into it.

~~~
usgroup
It reminds me of the “why we switched from Mongo to X” or “why we switched
from Ruby to Y” trends some years ago.

I saw Clojure become popular long before it was ready for the limelight: it
definitely wasn’t pragmatism driving it. It was a fetish for syntax and
paradigm.

------
didibus
It is interesting to read the history of Storm from the original author
(Nathan Marz) himself: [http://nathanmarz.com/blog/history-of-apache-storm-
and-lesso...](http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-
learned.html)

Clojure allowed him to build the initial Storm release by himself in only 5
months.

> I made all of Storm's APIs in Java, but implemented Storm in Clojure. By
> keeping Storm's APIs 100% Java, Storm was ensured to have a very large
> amount of potential users. By doing the implementation in Clojure, I was
> able to be a lot more productive and get the project working sooner.

Also interesting is that Nathan has now founded a startup working on something
which sounds like the next evolution of Storm, and he's building it in Clojure
again:

[https://redplanetlabs.com/](https://redplanetlabs.com/)

They recently got 5M in funding and are hiring. Building a Clojure team for
it.

Personally, I feel the Storm 2.0 release rewrite to Java is really just a case
of the new maintainers not knowing Clojure very well, and being primarily Java
devs, looking for contributions from people using Storm, which tend itself to
be mostly Java shops. Clojure is not quick to pick up, and very different from
Java, so unlike Scala, Kotlin, C#, Go, etc. Someone with a Java background and
no Clojure or Lisp experience won't be able to contribute as easily. And
that's fine, and possibly best for Storm's future now that Nathan has moved
on. Unless a Clojure dev had stepped up and adopted Storm, this change isn't
at all surprising.

------
rcaught
I struggle to see how you pick Java over Kotlin on the JVM these days. It's
everything that Java is, minus everything Java shouldn't be, plus everything
that Java should be.

~~~
ivan_gammel
Syntactic sugar does not solve programming problems. Kotlin does not offer any
paradigm changes which could make switch worth it.

~~~
vbezhenar
The only Kotlin feature that it's really hard to live without is properties.
Otherwise sure, it's better than Java 11, but I can live without all those
niceties. And (non)nullable types even get in my way!

I just don't understand where there's syntax sugar for propeties in Java.
There are lot of unneeded stuff added, that I don't really care about. But my
classes are still full of autogenerated getters/setters. I recently tried to
find some JEP and I did not find one. There was some talk around Java 7, but
nothing since that. Nobody even wants to add properties support for Java?

~~~
oweiler
Probably because in general they are a code smell. They are anti-OO.

~~~
vbezhenar
That's just not true. If I'm writing simplest REST service, I have to return
JSON responses. To return JSON response, I have to write class corresponding
to that response. And that will be data class. If I'm interacting with
database, I'm writing data mapping classes. And that will be data class. Those
are normal code that every developer writes every day. If you think that it's
anti-OO, then I don't care about that OO. I care about writing performant,
concise and extensible software.

~~~
ivan_gammel
That statement is valid in both contexts you mention. You do not need
properties for data mapping and serialization - it’s possible to write a good
OO code without getters and setters.

------
zenlot
_" While Storm's Clojure implementation served it well for many years, it was
often cited as a barrier for entry to new contributors. Storm's codebase is
now more accessible to developers who don't want to learn Clojure in order to
contribute."_ \- interesting phrasing. Seems that the authors assuming that
new contributors are more willing to learn Java instead of Clojure. And those
who know Java find Clojure difficult to learn? Clojure, in fact is quite easy
to grasp - especially for someone coming from Java, as you would still be
working on JVM and always have Java Interop. I guess same thing goes to
someone coming from Erlang to Elixir, where you are still working in the
familiar ecosystem.

~~~
cygned
> Clojure, in fact is quite easy to grasp - especially for someone coming from
> Java

I disagree on that point. In my experience as a programming language teacher,
people often struggle to learn functional programming and even more to learn a
Lisp, especially if they are used to a C-like language. And learning both at
the same time is even more difficult. I liked working with Clojure but I have
yet to meet someone who easily grasps the language and - more important -
development patterns.

~~~
adz5a
Hello, this is an interesting take. Clojure has been a hobby of mine for some
time but now I am a professional Clojure developper (clj/cljs). I knew the
language before starting my new job but I found it really different than know
the "platform": how a project is layed out, tests are written, some libs, how
to pinpoint potential technical debt/suboptimal code accurately and so on...
But I guess this would have been true if I had transition to any other new
language.

Would you have any example of what your trainees stumbled upon or found
difficult in the language? That would be useful I guess when we try to onboard
new people on the project.

Have a nice day :)

------
readme3
Apart from the language part , i am super excited about trying storm-2.0. In
many ways the buzz in the ecosystem has been that storm was dead!. We run lot
of prod workloads on storm and spark , storm has worked like a wonderful
workhorse without quirks. The ecosystem has moved on , but i think storm may
still continue having loyalists!

~~~
scribu
> The ecosystem has moved on

Just ouf of curiosity, what tools are people using instead of Storm?

~~~
tanilama
Flink is a strong competitor. Spark itself also claims to be a runner up with
minibatching.

But streaming model is more widely adopted outside of frameworks. AWS Lambda
or Fass platform can be viewed as streaming model with simple topology.

------
hanswesterbeek
Clojure just seems "too weird" to too many people, albeit for all the wrong
reasons. Its syntax looks like a 6ft hurdle to newcomers, especially if they
have lots of experience in C-synthax-languages. But anyone who just /tries/ to
jump it finds out the hurdle really isn't that high at all.

This dynamic will probably never change. Clojure will remain an acquired
taste, which is fine, I guess. Those 'in the know' will happily continue to
develop effectively with it.

------
keymone
Bet they had to reimplement half of Clojure when rewriting storm core.

~~~
juskrey
Yep, at last a lot of progress to report to Apache Board of Directors

------
mrkeen
In this thread:

* Non-Java JVM languages are too different for people to use.

Also:

* It's not worth switching from Java to other languages because they're just Java with syntactic sugar.

~~~
skywhopper
Clojure and Kotlin are two very different languages. I don’t see anyone making
the broad statements you are calling out.

~~~
mrkeen
No, of course not.

I misread

> Syntactic sugar does not solve programming problems. Kotlin does not offer
> any paradigm changes which could make switch worth it.

and

> Also, java 8 and above is getting a lot leaner to use, api, linguistics ..
> so the gap with clojure is a bit less than before.

------
juskrey
Apache products have always been a result of a blind majority rule, and thanks
to that many of them are a corporate-style incomprehensible mess. They favor
the process instead of a result: Apache things need to show constant progress
to be afloat.

Indeed, adopting Clojure for the developer product of this style is a dead
end. Clojure is a minority rule. A slow tinkerer's and a lazy (in a good
sense) practitioner's tool, whose aim is to survive in a real world, not to
show a steady movement no matter what.

~~~
Shmebulock
These Talebian references got me thinking. Couldn't you argue that it shows a
certain arrogance and non-results oriented "falling in love with ideals and
tools" detachment from reality to use Clojure instead of good ol' Java?

~~~
TeMPOraL
> _Couldn 't you argue that it shows a certain arrogance and non-results
> oriented "falling in love with ideals and tools" detachment from reality to
> use Clojure instead of good ol' Java?_

Since when trying to do better than using a tool designed for mass-producing
software with dumb and cheap labor is "falling in love with ideals and tools"?
It's kind of like saying that using an excavator over a bunch of people with
shovels is "falling in love with ideals and tools". No, it just allows a
single person to get more work done faster and better.

Our industry is weird. When recruiting, companies claim they want "the best of
the best" and will often test you on ridiculous stuff. But when it comes to
actual work, on industry scale, learning, growing professionally, and doing
things in a clean and efficient way is frowned upon. Best take the weakest
tool available and compensate for its shortcomings with third party services.

~~~
enlyth
> trying to do better than using a tool designed for mass-producing software
> with dumb and cheap labor

This is completely disregarding the available developer pool of Java
programmers, and the fact that a lot of them are really talented and can write
efficient, production ready software.

The fact is, there is at least an order of magnitude more developers to choose
from compared to Closure, and hiring and replacing people is hard. Ignoring
business realities like this is bad, and akin to "falling in love with ideals
and tools".

Also, someone using Java does not automatically make them dumb cheap
programming labour, same as someone using Closure is not automatically going
to be a 10xer who has memorised Knuth. It's possible to write bad FP code as
well.

~~~
TeMPOraL
> _Also, someone using Java does not automatically make them dumb cheap
> programming labour, same as someone using Closure is not automatically going
> to be a 10xer who has memorised Knuth._

I'm not claiming that. I'm claiming that Java has a long history of enabling
factory-farming software development, whereas Clojure doesn't lend itself to
this style. I mean that in the sense that an army of cheap labor with shovels
_can_ , in principle, do the same job as a few trained operators in
construction machines. The physics of dirt is the same, and so the job done is
essentially the same - just some tools let you do it faster with fewer people.
And then, the quantity (or lack of it) has a quality of its own - the more
people you engage doing smaller and smaller pieces of work, the more your work
becomes about coordinating people than doing the actual job.

(I'm aware that since version 8, Java is growing to be a quite decent
programming language. This somewhat weakens my criticism, but not all that
much.)

But yeah, I'm also claiming that ceteris paribus, the pool of Clojure
developers will yield you higher-skilled programmers on average than the pool
of Java programmers - simply because everyone and their dog knows some Java
nowadays, and some Java is enough to make some progress in factory-farming
software development, whereas Clojure is somewhat atypical and requires
expanding your competences beyond the very basics.

(Or, in other words, I'm claiming that the distribution of skill of Java
programmers is wider than that of Clojure programmers, with the latter having
higher minimum and mean values, and comparable max values.)

~~~
0815test
Does Clojure lend itself idiomatically to _any_ style other than quick-and-
dirty prototyping, though? There are real benefits to many features of Java
when it comes to long-term maintainability, even though it also has some very
real pitfalls at the same time (no optional values, no variant records with
pattern matching, no composition-plus-delegation-over-inheritance etc. etc.).

~~~
yogthos
Yes, it absolutely does. My team has projects that have been running for many
years in production. They're far easier to maintain than equivalent Java
projects we've done before. And there are plenty of companies, like Circle CI,
who built there stack on Clojure out there. Here's a talk from Kira Systems
who went from a quick and dirty startup to a 150 employee company in 8 years
using Clojure:
[https://www.youtube.com/watch?v=u-4FiFpkPlQ](https://www.youtube.com/watch?v=u-4FiFpkPlQ)

~~~
cutler
Considering Kira Systems had to resort to using Go for much of their heavy
lifting, due to Clojure immutability memory overhead, I would hardly concur
that this is a shining example of the benefits of using Clojure.

~~~
yogthos
Over 80% of their codebase is in Clojure, but obviously it's providing them no
benefit.

------
agumonkey
poor lisp, just like reddit ... always dropped for another language.

~~~
fulafel
The SW world is full of Java/C++ rewrites of products that originally started
out in a dynamic & productive language like Lisp or Python. It might make
sense in the lifecycle phase when you're past making big changes under make-
or-break pressure, and are in the "provide predictable 10 year maintainance
roadmap for this large codebaes" stage. You end up employing different kinds
of programmers too because many of the original team may be reassigned to
something more exciting/important to do.

But it's in no way inevitable, of course, there are many old Lisp, Python, etc
codebases in production too.

BTW. I've been trying to rediscover this blog post and can't seem to find it
again, anyone? It described different kinds of software projects, the (1) kind
that try to deliver incremental improvements (eg 10% cost savings), and the
(2) kind that try to make 10x improvements, and the difference in how those
projects are run (the 1-type project can't go over budget because it will then
deliver negative value etc).

~~~
0815test
Hopefully we'll start seeing some Rust rewrites in the future then, since the
pitfalls of C++ are widely-known by now, and few people would claim that they
_don 't_ impact long-term maintainability. (And the Rust Evangelism Strike
Force would surely approve of such a shift!)

~~~
geodel
If Rust Evangelism Strike Force volunteers to rewrite code in Rust instead of
heckling original developers to do it they may prove themselves to be more
productive and useful.

------
amortize
> While Storm's Clojure implementation served it well for many years, it was
> often cited as a barrier for entry to new contributors. Storm's codebase is
> now more accessible to developers who don't want to learn Clojure in order
> to contribute.

It is interesting to contrast this with state of affairs in Apache Spark.
Spark has thrived well in spite of being a Scala project; Scala arguably has a
higher barrier for entry compared to Clojure (although the flavour of Scala
used within Spark closely resembles Java).

------
grego
In the old days a project would often start in Lisp (Common Lisp for example),
and then later be rewritten in C.

This is the same pattern, and one should embrace it. Write the first cut,
explore the problem space and prototype in a dynamic language. Once the design
is proven, rewrite in statically typed language for performance and to get the
possible remaining bugs out.

If the design was perfect from the get go, you could just write it in
assembler to start with.

------
fulafel
For anyone interested in the Clojure history, there's this interview of Nathan
Marz, originalcreator of Storm, from 5 years ago:
[https://www.infoq.com/interviews/marz-lambda-
architecture/](https://www.infoq.com/interviews/marz-lambda-architecture/)

------
ww520
While most comments touch on the developer issue, the significant performance
gain deserves more discussion. I guess Java's static type, pragmatic
mutability with less garbage produced, and efficient parallelism give
substantial boost to performance.

It would be interesting to see what the upcoming blogs have to say about the
performance gain.

~~~
yogthos
My guess would be that it's always easier to make a rewrite performant because
you already have a working version and know exactly where bottlenecks are. I
imagine rewriting it in Clojure, or any other language, would've resulted in a
significant performance boost as well.

------
pjmlp
Sad to see it happen, however it is yet another example how platform languages
always win in the end, even if due to their nature and market size they tend
to be more conservative regarding feature adoption.

~~~
skywhopper
I dare say it may be _because_ they are more conservative regarding feature
adoption.

------
snorremd
> While Storm's Clojure implementation served it well for many years, it was
> often cited as a barrier for entry to new contributors. Storm's codebase is
> now more accessible to developers who don't want to learn Clojure in order
> to contribute.

While I appreciate that programmers have limited free time and may not want to
learn a new language to contribute to open source, I feel like learning
Clojure could be of value to the programmer in itself.

The supposition that Clojure acts as a big barrier against contribution from
developers might be true, but I really feel like developers who only write
code in the ALGOL family of languages are missing out. Learning a lisp
language is not an impossible task and can improve your skills as a developer.

------
didibus
Can someone enlighten me on the inner working of Apache?

Do they hire developers to maintain and work on the projects? If not, is there
community leads in charge of each one? How are they appointed?

~~~
geodel
[http://www.apache.org/foundation/governance/](http://www.apache.org/foundation/governance/)

~~~
didibus
Interesting, where would you go to see the list of committers for a given
project and find information about them?

I'm curious if the commiters are Clojure developers who chose to switch to
Java, or if they are Java devs who decided to rewrite the project to a
language they are more familiar and comfortable in. Or if they are Clojure
devs looking to transition the project to other commiters and they struggled
to find other Clojure devs willing to take it over.

------
zubairq
We did the same with our open source project. We ditched Clojure and
Clojurescript for NodeJS and JavaScript, for the same reason as it was a huge
barrier to entry for new users

------
gigatexal
How am I just learning about this? This seems to be rather novel. Real-time
bolt on for any database? Pretty cool stuff.

------
grkg8tr
How does this relate to Apache Heron? I thought Heron was essentially Storm
2.0. Which one should I be using?

~~~
roshan_naik
Not well known is that Heron's performance claims in the "flying faster" blog
and SIGMOD paper were made by comparing against a fork of an older (pre
apache) version of Storm even though much more performant version was already
out.

IMO (biased?) Heron makes surprisingly poor design choices for their key
distinguishing features like threading model and backpressure. And it shows
when you benchmark it. More specifics:
[https://github.com/apache/storm/pull/2241#issuecomment-31787...](https://github.com/apache/storm/pull/2241#issuecomment-317879907)

------
jaimex2
If you're going to be on the JVM, stick to what the J stands for :)

~~~
lvh
The newest JVM out of Oracle, GraalVM, has its polyglossy as its central
feature. And Clojure has always been a lot closer to Java (in terms of quality
of interop) than even e.g. Scala, let alone something like Jython. (Notable
exception JRuby.)

------
jacobtracey
What exactly are you doing?

------
fulafel
cutting off 1.0 immediately sounds quite harsh, I wonder what kind of user
base it has?

"EOL for 1.0.x With the release of 2.0.0 the 1.0.x version line will no longer
be maintained."

~~~
Hendrikto
> It is designed to push boundaries on throughput, latency and energy
> consumption _while maintaining backward compatibility_.

------
kabhwan
DISCLAIMER: I'm one of PMC member of Apache Storm.

Even though Clojure has been Nathan's first-class language, not everyone in
contributors and even committers couldn't become native on Clojure. (Though I
guess part of committers could feel Clojure as native.) It might not be a
problem when Nathan builds most of things in Storm (I'll link his blog post
which describes lesson learned from Storm on his perspective
[http://nathanmarz.com/blog/history-of-apache-storm-and-
lesso...](http://nathanmarz.com/blog/history-of-apache-storm-and-lessons-
learned.html)), but when Storm becomes popular and also no longer his toy
project, Clojure matters for scalibility of the project.

Personally, I had a hard time understanding plenty of macros and even chained
macros - that might not be a problem if I were allowed to have plenty of times
to study Clojure and get used to it, but possible contributors don't get a
chance when they're ready. In many cases, the first time they read the code on
what they're using is when they encounter issues on stage/production and have
to dig with stack trace. For me, I had to investigate on Storm because my team
suffered from lack of documentation and also lack of knowledge on internal
which ended up just taking workarounds. It was pretty hard time learning
language to just read and understand the details to track down the issue. I
even couldn't think of writing code on Clojure.

When I decided to contribute Apache Storm, I started from connectors and CI
build which wasn't written on Clojure. After some months I was able to read
and understand the code to get into details but I still couldn't be native for
Clojure, so still required N times of efforts to read and write the code even
after I became one of PMC member.

Don't get me wrong. I'm not saying something is better than some other one. In
scalability point of view on project, I guess we had to do this, even though
it ended up teaching me some lesson that migrating language is not the thing
which can be considered easily, especially for a big project. Lots of efforts
were spent here.

Someone could claim for Apache Spark's case for Scala. Yes, I'm also
contributing to Apache Spark and I had to learn Scala (fortunately I had a
chance to do that before contributing) but the case is different. If you read
Databricks Scala guide in details, you might realize that the doc is actually
not encouraging contributors to use advanced features (a.k.a the features only
Scala geeks might feel natural) on Scala. Same goes on reading codebase.
Apache Spark community has been trying to keep necessary knowledges on Scala
to contribute Apache Spark low enough, so that non-expert of Scala engineer
could easily try out contribution. If they have been requiring hard
understanding of Scala to accept contributions - Spark may not be able to get
its amazing reputation.

------
_cs2017_
Why change the title of the post?

Almost every one of the ~100 comments is in response to the original title
that Storm switched from Clojure to Java. Now this context is lost.

~~~
lvh
Because the previous title was editorialized? Furthermore, perhaps the
conversation is about the old title because it was, and now the conversation
can go be about Storm itself :)

~~~
_cs2017_
I think the rule about not editorializing needs an exception so that someone
can point out an important or interesting fact using an article whose original
title is too generic.

Another example would be "Press release from XYZ". If it talks about XYZ
reaching some major breakthrough, the title should be allowed to reflect that.
It's a waste of everyone's time to post the article titled "Press release".

And arguably this situation should fall under such an exception.

------
ian1321
Java is statically typed. Clojure is not. To me, this is the biggest
difference and the reason to choose Java.

~~~
jnwkw
Is this true or false? I don't understand the downvotes.

~~~
Rovanion
While it is true that Java is statically typed and Clojure is dynamically
typed it is perhaps one of the less important differences between the two
languages. If you in fact would like to add a static type system on top of
Clojure there is a way to do that: Typed Clojure.

[https://typedclojure.org/](https://typedclojure.org/)

