
Clojure at Netflix (2013) [slides] - tosh
https://speakerdeck.com/daveray/clojure-at-netflix
======
codetrotter
Would be interested to hear their experiences with Clojure at Netflix for the
5 years that have followed since then.

Did they keep writing more Clojure? If so, how much of their code is now in
Clojure compared to Java? How much more did they rewrite from Java to Clojure?
Do they use Clojure rather than Java for new code?

What other languages do they use? Python? Erlang? Rust? How much, in terms of
functionality, is written in each language they use? How many lines of code do
they have in each language? To what extent did they find existing libraries
for the different languages usable for what they are doing?

Among the things that seemed great with Clojure in 2013, did they find that
some of these were not so great after all once the codebase grew? Any other
problems?

~~~
diab0lic
I am very glad you asked!

I wrote and deployed (to production) some Clojure code at Netflix just
yesterday. Among other things at Netflix the Mantis Query Language (MQL an SQL
for streaming data) which ferries around approximately 2 trillion events every
day for operational analysis (SPS alerting, quality of experience metrics,
debugging production, etc) is written entirely in Clojure.

This runs in nearly every critical service, ~3000 ASGs and easily > 100k
servers and Clojure allows us to also compile it for our NodeJS services as
well.

~~~
shadowmint
Thats an interesting point that doesn’t answer any of the questions the parent
asked. :)

Perhaps you can answer this simpler one:

Netflix has been using clojure for a long time now; has that been a positive
experience broadly speaking, that means clojure is still being used for new
projects, or not?

Having a large successful project in clojure is lovely, but much of the
community’s concern around it is that its hard to maintain, and falling in
popularity, broadly speaking.

It would be very nice indeed to see _those_ points addressed by a large scale
user of clojure.

~~~
diab0lic
Hey, sorry I saw that and typed a quick response just as I woke up. I'm not
usually at a computer so early in the day. I'll address these now that I'm in
front of a machine. :)

> Did they keep writing more Clojure?

Yes but it has never been the primary language at Netflix.

> How much more did they rewrite from Java to Clojure?

Very little, if any was rewritten from Java.

> If so, how much of their code is now in Clojure compared to Java?

A very small amount given that it isn't the primary language and Clojure code
bases tend to be much smaller than Java.

> Do they use Clojure rather than Java for new code?

This is a personal choice each engineer makes when they write new code. Those
who like Clojure might reach for it more often. Clojure is also easy to use
within the environment at Netflix since everything was JVM based.

> What other languages do they use? Python? Erlang? Rust?

NodeJS and Javascript, Python, Ruby all have a seat at the table but the
majority of back-end code at Netflix is on the JVM, the majority of that is
Java.

> Among the things that seemed great with Clojure in 2013, did they find that
> some of these were not so great after all once the codebase grew? Any other
> problems?

I've always found larger Clojure code bases to be a bit unwieldy. Fortunately
you can usually continue to abstract and keep the size small. If you choose
your abstractions carefully you can get a lot of mileage out of this.

I've found the lack of static typing to be a bit of a pain at times especially
when refactoring. My safety net for this in the project mentioned in the GP
post is to have comprehensive unit tests. If I were to initiate this project
today I'd likely explore using Spec to make type assertions.

~~~
vram22
>Clojure code bases tend to be much smaller than Java.

What are the reasons for this? FP language vs. OOP? Less boilerplate (again
maybe due to FP)? Higher-level abstractions in the language or libraries?

I have seen that F# code (another FP language, although I've read F# is more
from the ML family via OCaml, vs. Clojure being from the Lisp family) can be
significantly shorter than equivalent C# code, for example, as shown in some
comparisons on the fsharpforfunandprofit.com site.

Interested to know.

~~~
wild_preference
I worked at a shop that used Clojure and Java.

One big difference is that Java APIs tend to require the collaboration of
various class instances to get something done, things that you would implement
as a single function + options object on your own.

Bouncy Castle is a good example. You may need a Hasher, HasherStrategy,
ASNEncoder, DERParameters, and ASNSerializerStrategy instances to execute what
you would've implement as `(asn-encode thing)` otherwise, maybe even having to
subclass some of them to change some behavior you'd expect an option flag for.

Clojure's own Java-helper macros will compact your 1:1 Java interop code as
well, so you have fewer lines even when writing Java from Clojure. Also,
short-cuts like ad-hoc reification in Clojure will spare you LoC where you
might otherwise have created a whole file for a class with custom interface
implementation in Java.

Of course, line-to-line code is also just more compact in Clojure, but
ecosystem/api difference is one I don't see mentioned as often.

I don't think this is just good vs bad, though. There are certainly upsides to
the more rigid everything-in-its-right-place code you tend to have in Java
which has been making big strides in improving itself over the past decade.

~~~
vram22
Interesting, thanks.

I agree with your last point.

------
lvh
Latacora has its token Clojure person: me. I write Clojure every occasion I
get, we've shipped Clojure code to customers, et cetera et cetera.

Most of the stuff I'm doing in Clojure would be possible in other languages,
but in a sense that's only one notch below "Turing tarpit" possible. So, in
the only sense that a startup should care (would you f'ing ship it?): only
possible in Clojure or a handful of close cousins.

I'm hearing less about it on HN these days? But it was a great language
yesterday, remains one today, and there's constant development of tooling and
community :)

~~~
jwhitlark
I think some of that is due to the nature of the people it attracts, namely,
grumpy, older, I've been burned before, programmers. I use Clojure constantly
and have for 10 years, but I've never felt a strong need to proselytize. I
just make money and enjoy coding, day in and day out.

------
m12k
I'd like to learn a modern production-ready functional language (I have some
academic experience with SML), since they seem like a good way to grow as a
programmer. Main contenders so far are Haskell, Scala and Clojure (Reason
might go on the list soon too) - but the fact that Clojure is dynamically
typed is a bit of a turn off for me. My experience with dynamic vs static
typing is that as a system grows in size and complexity, the value of static
typing goes up. Doing a major refactor is one of those things that I find
especially error-prone in dynamic languages, while statically typed languages
make this much less daunting (of course they come with the opposite issue that
you might not be able to compile for a day until you've fixed all the type
errors - but I'd take this trade-off any day of the week). Can anyone with
experience comment whether Clojure suffers from this too or if it somehow
handles the lack of static typing better than other dynamic languages?

~~~
blandflakes
There are a lot of people who claim they're more productive in Clojure, in
spite of or because of its dynamic typing.

I'll just throw my anecdatum out here: I found that dynamic typing was a major
pain even on my own personal projects. I also never found REPL-driven
development to mesh well with my workflow.

In Scala, for example, entire classes of error that I just shouldn't be able
to make don't exist. Maybe I'm just the kind of person who works better with
static typing.

Clojure overall is an excellent language with really good features. It just
won't get out of my way sometimes.

~~~
xapata
Try to grok REPL-driven development before dynamic typing. You won't like the
latter until you love the former.

~~~
jerf
Haskell does REPL-driven development fairly well. It's not the _best_ REPL,
but it works, especially if you take a moment to learn a few of the commands
to drive it. Dynamic types are not necessary for REPL-driven development.

~~~
xapata
Yes, that's why becoming happy with REPL-driven development is more important
than becoming happy with dynamic typing. I think much of the productivity
boost attributed to dynamic typing is in fact caused by heavy use of a REPL,
and not dynamic typing.

~~~
blandflakes
I'd maybe consider that perhaps becoming happy with REPL-driven development
isn't a universal solution. It's not like I didn't understand the concepts,
the tools, or the process. I just didn't find that it made me any more
proficient at building software.

~~~
xapata
Of course that's possible, but the evidence suggests that most people become
more productive when using the REPL to experiment while developing large
projects.

One issue with trying to adopt the habit midway through a project is having a
design that makes it easy to load small portions of the project with test data
into the REPL. It's not quite the same as having code that's easy to test.

Do you use an interactive debugger? It's the same principle, but earlier in
the lifecycle of development.

~~~
blandflakes
I appreciate your earnest approach, but I also feel somewhat patronized. I've
been doing this for a while. I used Clojure extensively for years, was
productive and found much to like. The REPL wasn't part of it, and I don't
miss it. I still use Clojure on occasion.

This response is pretty common: "You probably weren't doing it right. Trust
me, this is the way!" While I agree that suspending beliefs/old practices and
trying new things is important when using new technology, I'm also pretty
confident that I gave Clojure a more than fair shake, and just find that I
don't get much out of the technology surrounding the REPL in that environment.

I was more productive in Scala in days than I was after years of using
Clojure, and the difference between what I'm saying and your response is that
I'm not claiming my experience is universal.

~~~
xapata
No condescension meant. It's hard to estimate how experienced someone is from
a few sentences in the forum. I'm generally careful to make assertions about
_most_ and _in general_ rather than _all_ or _universally_.

~~~
blandflakes
None taken personally - I just find it valuable when I read comments against
the grain, and had an experience that I know many Clojurists find atypical and
wanted to share that my style of work just doesn't lend itself to Clojure as
supported today.

------
nerdponx
What happened to Clojure? Was it just a fad, or are people still using it in
their day jobs? Are people still hiring Clojure devs? If you write Clojure at
work, are you happy with it?

~~~
swah
While other posters comment that "Clojure still works", IIRC the "promise" was
that Clojure/Lisp would be _much, much more productive_ , and _that claim_ I
feel like must been disproved in daily work, otherwise Clojure would still a
main topic around here.

Reminds me of the classic Erann Gatt post:
[https://news.ycombinator.com/item?id=2308370](https://news.ycombinator.com/item?id=2308370)

(I loved to play with Elisp, Clojure, Lua, Factor, Ion, Ioke and hope dynamic
languages become fashionable again someday...)

~~~
empthought
I think _individual developers_ are most likely to become _much, much more
productive_ when using Clojure/Lisp vs. Java, C#, and the like, but _software
development teams_ are unlikely to experience the same benefits.

And it's not even because Clojure is bad for teams, it's because teams often
form because there are nontechnical stakeholders, and suddenly an individual
contributor's code-writing productivity is a much smaller factor when compared
to the communication practices and other parts of the development process
outside of writing code.

------
projectileboy
This is a bit tangential to the post, but I've seen a couple comments in this
thread that amount to "Clojure is hard to read", but I think that's a really
unfair comparison, because people are unwittingly comparing apples and
oranges. It seems to me the reason people think Clojure is hard to read is
that the language is so powerful and expressive, that when you're reading
Clojure code, you're typically trying to understand the actual problem that's
being solved, as opposed to just reading a bunch of code that's necessitated
by the lack of power afforded by the language. For example, I can easily
"read" a bunch of nested for loops in Java, but then I still have to try and
understand what the actual problem is that's being solved. People seem to
confuse those two things when comparing Clojure with other languages.

~~~
ronnier
I worked in a large company where clojure was used. After the original
developers moved on from that project (they always move on), it was a giant
struggle to get people who could work in the code base. They ended up
rewriting it in something more standard. Whatever benifit clojure provided was
outweighed by the high cost of maintaining the code base and the inability to
find devs who wanted to work on it.

That’s my only data point with clojure.

~~~
TeMPOraL
It saddens me to hear such stories as they show up from time to time. I can't
help but feel that a lot of programmers just decide to stop learning at some
arbitrary point in their career (my observations suggest that point is around
the end of their university term or their first job). The industry too seems
biased heavily in favor of using lowest-common-denominator, dumbest possible
solutions they can get. And not "dumbest" in the sense of "simple therefore
robust" \- but in the sense of "barely suitable for the job at hand". It's
like wanting to dig a pool, and hiring a lot of people with spades, even
though you can get excavators for free - but what, you couldn't be bothered to
find a few drivers? And the spade crew comes and is all, "no way we're going
to learn to operate an excavator, that's too much, let's dig by hand".

I can't explain it any other way. Learning Clojure, or another Lisp, to the
point of being comfortable around a codebase, is a matter of weeks for an
experienced programmer. It's comparable to the time you need to even figure
out how to rewrite an existing mid-sized codebase, and since you have to read
it to understand it, why not learn the language along the way? Then suddenly
the rewrite isn't needed. But it feels like a concept of learning a new
language on the job isn't even considered by people.

~~~
lliamander
> I can't help but feel that a lot of programmers just decide to stop learning
> at some arbitrary point in their career (my observations suggest that point
> is around the end of their university term or their first job).

Indeed.

I had the frustrating, but ultimately fortuitous, situation of having my first
job be one where I was asked to build a solution with woefully under-powered
tools. I'm not even talking about the difference between hand tools and power
tools - I was essentially asked to use kid toys.

That situation made me very angry, and I was constantly working overtime to
make up for the lack of productivity of my tools - even then we were going to
miss our deadline.

Finally, I managed to convince management to let me and my team use real tools
- nothing super advanced, but well suited for the job. Very quickly we turned
around the release, I started working normal hours, and when I left that job
shortly after the release my employer found that they didn't need to hire a
replacement because the workload was so much less.

This experience taught me that the power of your tools _does_ matter, and that
the initial pain of technology transition can easily be outweighed by the
benefits of using something better.

For me, the quality I look for in my tools is: does it reward mastery? I'm not
(primarily) looking the initial growth curve and how long it takes before I
get to my current productivity. Rather, I'm looking at where the ceiling is.

------
tosh
via tonsky:
[https://twitter.com/nikitonsky/status/1057317735766593536](https://twitter.com/nikitonsky/status/1057317735766593536)

    
    
      - Migrate as you go
      - 1/10th the code that does 10× as much
      - Life with existing Java codebase is better with Clojure macros 
      - Eliminate boilerplate with custom DSLs
      - REPL to live servers

------
mikeflynn
We have used Clojure in our data processing and API code for five years now at
Studio71, and it's been a huge success for us. We have much smaller dev teams
than our competitors so Clojure has helped us keep pace with a smaller code
base and fast data processing, but it has been a challenge to hire new
developers when we need one, but with patience it has always worked out.

------
auganov
This talk goes into detail how Clojure plays out in larger teams, particularly
how they worked to maintain REPLability for a huge monolith:
[https://www.youtube.com/watch?v=BThkk5zv0DE](https://www.youtube.com/watch?v=BThkk5zv0DE)

That said, I think a lot of the preoccupation with rewrites is just premature
optimization. Just a while ago I noticed some open source Clojure code of a
startup acquired by FB got rewritten in Haskell. Was very glad to see founders
don't seem to be working on this new code-base. Probably moved on to do higher
value things.

~~~
truculent
FWIW I think this is the startup code referenced: [https://github.com/wit-
ai/duckling_old](https://github.com/wit-ai/duckling_old)

~~~
auganov
Yep, that's the one

------
TheMagicHorsey
I love Lisp, so I thought Clojure would be a great productivity booster. And
it is, if you are a one-man or one-woman shop. But try to build a team around
a Clojure project and it's another story. The language basically begs you to
make "magic" happen with domain-specific constructs. That might make you feel
powerful as a programmer, but it's also a nightmare for new team members to
get up to speed on.

And then, as others have mentioned, it's hard to find Lisp programmers. Most
programmers are turned off by the syntax and/or immutability. I happened to
take a Lisp class as an undergrad, but I would otherwise probably be in the
same boat.

~~~
jimbokun
"Most programmers are turned off by the syntax and/or immutability."

Programmers who don't see the value of immutability or can't understand it are
not programmers you really want to work with.

~~~
jcadam
> Programmers who don't see the value of immutability or can't understand it
> are not programmers you really want to work with.

I'm not disagreeing with you, but since you just described easily 90%+ of all
working programmers, it's kind of unavoidable.

One place I worked, one of the lead engineers knew I was into functional
programming and asked me if I thought it would be a good idea to introduce
Scala or Clojure into our pure Java code base. I said "No. The majority of the
programmers here couldn't make the adjustment."

Put another way, if I could assemble a hand-picked team of all of the
engineers I would rate as "great" who I've worked with over my 12+ year career
thus far -- I'd have a team of about 7 people.

------
rb808
Is Clojure still a thing? It seems to have fallen off my radar. Java seems
resurgent after 8, even Scala seems to have lost momentum.

~~~
systems
to talk about something new, you need to have something new to talk about

i dont think that scala or clojure have lost momentum i think both clojure and
scala, stopped adding new features that are worthy of blogging about

you cant just add feature, just to keep people talking about you

clojure, is what it is, a nice language, a very smart leader (rick hickey),
and a nice vibrant community (check #clojure on twitter)

the most talked about language nowadays is actually c++, and i think this
mainly because they suddenly decided to have a very ambitious roadmap, and
keep adding features, so there is always something new to talk about

does this make c++ better than clojure, well, i dont think they are even
comparable, but it just make c++ more talked about than clojure

~~~
kamaal
>>i dont think that scala or clojure have lost momentum i think both clojure
and scala, stopped adding new features that are worthy of blogging about

More like Clojure is in Cathedral mode of open source development. Most of the
development is in private discussions, implementation and prototyping at
Cognitect, and once they have something to share, they release it to the
world.

That's more a communication and marketing thing. For example, you can try
Googling but you won't be able to see anything on the Clojure roadmap. No one
knows what's the plan even year or two from now.

But thanks to smart leadership, and brilliant development team they have, its
mostly a non issue if you don't really care about the everyday minutiae. You
can bet the top dollar that they are working on something good, and it will be
eventually released to the world.

To people coming from other open source tools, that can sound as strange.

~~~
puredanger
[https://clojureverse.org/t/roadmap-
to-1-10/1581/8?u=alexmill...](https://clojureverse.org/t/roadmap-
to-1-10/1581/8?u=alexmiller) is a pretty good overview of where Clojure has
been going and will continue to go.

------
finalfantasia
As the leader in online affiliate marketing, we’ve just launched our analytics
and insights platform that is written in Clojure [0]. If you like Clojure,
come work with us [1].

[0] [http://junction.cj.com/article/introducing-insights-cj-
affil...](http://junction.cj.com/article/introducing-insights-cj-affiliate)

[1] [https://engineering.cj.com](https://engineering.cj.com)

------
vbuwivbiu
amazed to see there are people who find

myObject.myMethod(x, y, z);

easier to read than

(my-function x y z)

\- that's 2 delimiters in Clojure vs 6 in your C-style language

~~~
zeveb
In fairness, it'd actually be:

    
    
        (my-method my-object x y z)
    

But yes, the Lisp version is so much clearer to me. I just don't get why it's
not more popular.

~~~
trevor-e
How does this work with autocomplete if you need to know the method name
before the object you're working with? Maybe I'm not getting something but
that sounds incredibly painful to use in practice.

~~~
zeveb
In Common Lisp, it works because symbols are scoped by package, not by object.
So if you're in the package foo and start typing 'bar,' then your editor would
autocomplete symbols in the foo package starting with 'bar'; likewise if you
typed 'baz:quux' your editor autocomplete would autocomplete symbols in the
baz package starting with 'quux.'

Common Lisp most definitely doesn't have one namespace; this is really nice in
practice.

~~~
trevor-e
OK so my confusion was from thinking Clojure is statically typed, which
apparently is not true. I was basically asking the same question as
[https://www.reddit.com/r/Clojure/comments/6f6cq2/how_to_comp...](https://www.reddit.com/r/Clojure/comments/6f6cq2/how_to_compensate_for_clojures_lack_of_dot/)

------
666lumberjack
Interesting stuff. I'd be curious to know how Netflix are getting on with it
now.

------
dr_teh
I love Clojure! It's honestly the best language ever.

------
truculent
I always see loads of Static vs Dynamic and Functional vs OO opinions breaking
out in the comments. While the discussion is often interesting and insightful,
it would be nice to see some empirical analysis of the trade-offs here. Stuff
like development time, error rates, maintainability (no idea how you'd measure
this one...).

Anyone know if such studies exist? Feels like they should be out there given
the amount of discussion the topics generate

~~~
yogthos
There is some research available on the topic. There's a good summary of it
here [https://danluu.com/empirical-pl/](https://danluu.com/empirical-pl/)

~~~
truculent
Lovely, thank you!

------
jimbokun
I would like to know the words spoken for this slide:

[https://speakerdeck.com/daveray/clojure-at-
netflix?slide=29](https://speakerdeck.com/daveray/clojure-at-netflix?slide=29)

------
systems
is clojure still used at netflix [2018]?

------
choot
One strategy which i learnt from a tech company executive is that if you want
to hire best of best at low cost, develop products in interesting new
programming languages as the most curious programmers who are very serious
about programming care less about money and more about work in a particular
language of their interest.

No wonder the company had successful projects in every asoteric languages.

Now, I come from executive background in corporate job where going with
anything other than Java is considered risky. This came off as suprise to me.

~~~
lemming
_...if you want to hire best of best at low cost, develop products in
interesting new programming languages as the most curious programmers who are
very serious about programming care less about money and more about work in a
particular language of their interest._

The problem is that the particular language of their interest probably changes
every 6 months.

~~~
narwally
>The problem is that the particular language of their interest probably
changes every 6 months.

As a language nerd, one of the nice things about Lisp is that the language is
easily extended into other paradigms. I've stuck with Clojure for a long time
because of this. It had Go style CSP-concurrency as the core.async library
within a few years of release. I can use types by just loading core.typed, or
I can play around with logic programming with core.logic. Whenever I find a
cool feature in another language, I usually find out that it's already
available in Clojure as a library.

