
Rust for Clojurists - jarcane
https://gist.github.com/oakes/4af1023b6c5162c6f8f0
======
moonchrome
I have this crazy idea that you could build Clojure on Rust the same way
Clojure sits on top of JVM right now and it could provide you with some
interesting benefits.

First of all Rust doesn't implement a GC but it has very precise memory
management semantics which means integrating in to a GC should be possible and
safe.

I believe Clojure immutable data structures would allow you to (theoretically)
build a fast/simplified GC because immutable data has some very nice
properties :

* no cycles except trough reference types which are explicitly marked - this means you never need to scan old generation to collect the new generation

* data is immutable which means copying can be done in parallel

* mutable references are explicitly marked and isolated which means you could have a very minimal pause to update references

And Rust could safely encode all those requirements in it's type system to
make the interop safe. No need to sprinkle write barriers all over the place,
localized and isolated mutable references (you can modify the update semantics
and track them easily), minimal pause times, parallel collection, etc.

I think there are other interesting things you could do, for example Clojure
is pretty static for a dynamically typed language and protocols map nicely to
traits I wonder if you could somehow remove the dynamic typing and just have
compile time type inference figure out the types (requiring explicit types
when the system can't infer the types) - this would allow you to send Clojure
down to LLVM along with Rust. Or maybe you could have an interpreter for the
truly dynamic stuff - but then you'd need to figure out how to export
metadata.

Also I'm sure that the only sane/safe way to do it would be to restrict the
kinds of objects and that would require serious thinking too.

~~~
pron
> this means you never need to scan old generation to collect the new
> generation

Even in HotSpot you don't need to scan the old generation if references in it
don't change. You only scan (roughly) those old objects who hold references
that have been mutated since the last collection.

> data is immutable which means copying can be done in parallel

This is also possible when mutation is allowed (see Red Hat's
Shenandoah[1][2], A new GC for HotSpot), and immutability doesn't make the
challenge significantly easier (there is added overhead for mutation only).

> mutable references are explicitly marked and isolated which means you could
> have a very minimal pause to update references... No need to sprinkle write
> barriers all over the place

Again, mutating reference fields has a special barrier in HotSpot's GCs, but
they're not "sprinkled all over the place" \-- they're as good as being
specially marked.

To summarize, current (and future) HotSpot GCs punish you for mutation, but
not for the _possibility_ of mutation; you get all the benefits of
immutability automatically when you simply don't mutate references.
Immutability -- as you note -- might indeed simplify the GC (though not in the
case of Clojure, which does allow quite a lot of mutability -- not everywhere,
but enough that you have to account for it not as an outlier), but it doesn't
make it faster.

\----

> for example Clojure is pretty static for a dynamically typed language and
> protocols map nicely to traits I wonder if you could somehow remove the
> dynamic typing and just have compile time type inference figure out the
> types (requiring explicit types when the system can't infer the types) -
> this would allow you to send Clojure down to LLVM along with Rust.

A good JIT (like HotSpot) does this a lot better. It is true that Clojure is
nowhere near as crazily-dispatched as, say Ruby, but it does rely _a lot_ on
Java-like interface-based dynamic dispatch. If your type inference is really,
really good, you might be able to identify a great deal of the monomorphic
call-sites, but a good JIT finds all of them and makes those calls virtually
free, something an AOT compiler can never hope to achieve. Besides, HotSpot's
optimizing JIT is a state-of-the-art compiler. You won't do any better with
LLVM.

There are approaches that are good for some languages, and others that are
good for others. Language's that rely on dynamic dispatch are better off using
a JIT than an AOT compiler (actually, pretty much every language would get
better code generation with a JIT -- except C which hardly does any dynamic
dispatch -- but a JIT has some runtime overheads in warmup, RAM and energy,
and languages that don't do too much dynamic dispatch can get excellent
machine code generated by an AOT compiler, so that a JIT isn't worth it).

\----

In short, while you could (maybe) compile Clojure down to Rust, you'll see no
benefit, and probably quite a few disadvantages to that approach. If you want
Clojure-Rust interoperability, you can get it today with RustJNI[3].

[1]: [http://openjdk.java.net/jeps/189](http://openjdk.java.net/jeps/189)

[2]: [http://www.jclarity.com/2014/02/19/shenandoah-a-new-low-
paus...](http://www.jclarity.com/2014/02/19/shenandoah-a-new-low-pause-
garbage-collection-algorithm-for-the-java-hotspot-jvm/)

[3]:
[https://github.com/Monnoroch/RustJni](https://github.com/Monnoroch/RustJni)

~~~
moonchrome
>Even in HotSpot you don't need to scan the old generation if references in it
don't change. You only scan (roughly) those old objects who hold references
that have been mutated since the last collection.

>This is also possible when mutation is allowed (see Red Hat's
Shenandoah[1][2], A new GC for HotSpot), and immutability doesn't make the
challenge significantly easier (there is added overhead for mutation only).

You pay for this with barriers and implementation complexity.

>To summarize, current (and future) HotSpot GCs punish you for mutation, but
not for the possibility of mutation; you get all the benefits of immutability
automatically when you simply don't mutate references. Immutability -- as you
note -- might indeed simplify the GC (though not in the case of Clojure, which
does allow quite a lot of mutability -- not everywhere, but enough that you
have to account for it not as an outlier), but it doesn't make it faster.

>A good JIT (like HotSpot) does this a lot better. It is true that Clojure is
nowhere near as crazily-dispatched as, say Ruby, but it does rely a lot on
Java-like interface-based dynamic dispatch. If your type inference is really,
really good, you might be able to identify a great deal of the monomorphic
call-sites, but a good JIT finds all of them and makes those calls virtually
free, something an AOT compiler can never hope to achieve. Besides, HotSpot's
optimizing JIT is a state-of-the-art compiler. You won't do any better with
LLVM.

Rust code doesn't pay for write barriers at all - my point isn't that Clojure
on Rust would be faster than Clojure on JVM, it's that Rust code allows much
more deterministic performance, better tools for manual optimizations and
native interop than JVM and you'd write the performance sensitive code in
Rust. Clojure is then a high level tool to consume Rust code.

I think you're looking at this the wrong way and missing the obvious benefits
- you would get a simple _almost_ pauseless parallel GC and a high level
language with trivial native/safe rust interop (which JNI isn't).

Because of Rust type system you could even isolate threads that have no access
to the shared heap and not collect/pause on those - so no chance at pauses at
all. And AoT compilation also gives you deterministic performance.

Anyway it would be a lot of work for sure, I doubt it would be a straight port
of clojure (would need to make it more static to fit to AOT nicely) and you
would have to modify Rust to generate metadata so that the dynamics parts
could run (or just ditch the dynamic part completely and force everything to
be resolved at compile time with a simplified/constrained type system that
would be simpler to infer without explicit typing).

~~~
pron
> you would get a simple almost pauseless parallel GC

I don't see why you think you'd do any better than HotSpot's GCs in terms of
pauses. You haven't described anything any of the current three production-
quality HotSpot GCs don't already do (except for concurrent copying, which
Shenandoah does). They also don't use write barriers -- unless you mutate
references -- and you'd need those barriers when you mutate in Clojure anyway.
I just don't see where the GC you've described differs from Clojure's current
GCs.

Also, I'm not sure why you think Clojure data structures contain no loops
except when relying on mutable references. Clojure sequences are just an
interface, and it's easy to construct a lazy seq that contains loops.

> my point isn't that Clojure on Rust would be faster than Clojure on JVM,
> it's that Rust code allows much more deterministic performance, better tools
> for manual optimizations

But _Clojure_ would take away from that deterministic performance _and_ from
your ability to manually optimize! Rust is a language designed to achieve good
performance -- with safety -- in _resource-constrained environments_. You then
want to take away that advantage by compiling a language that is both too
wasteful for such environments and not deterministic.

The JVM has its use-cases and Rust has its own, and Clojure is by far more
appropriate for the JVM use-case, so I still don't see the point. For
performance sensitive code in Clojure today, you drop down to Java (or
Kotlin).

Also, it is _extremely_ hypothetical that you could get a higher-level, less
hand-optimized language than Rust to enjoy Rust's benefits (that are bought
precisely by paying for them with a more complicated language). I don't see
how that could be possible with Clojure. You could, of course, create a more
Lispy syntax for Rust -- with Rust semantics and a GC -- but how useful that
language would be over, say, Common Lisp is unclear to me.

Think of Rust as a language that makes C++ safer, not as one that makes Java
faster/cheaper. Rust can't offer the same abstractions Java/JVM provide for
the same low cost -- it offers the same abstractions C++ does, with much
better safety. I think that mixing the two will just reduce their advantages.

> which JNI isn't

RustJNI is very nice[1]. Again, I don't see how you think to do better than
the current Clojure runtime, which is written in C++. Sure, it might be easier
to write in Rust, but the C++ code already exists, and man-centuries of effort
have been put into it. Clojure's heavy reliance on interface polymorphism and
garbage collection takes advantage of pretty much every optimization provided
by that runtime (and more are coming! see later).

> Because of Rust type system you could even isolate threads that have no
> access to the shared heap and not collect/pause on those - so no chance at
> pauses at all.

If that's so important, you can do the same with real-time JVMs (RTSJ). They
have zero-pause non-heap threads (in fact, they're called
NoHeapRealtimeThread[2]), and they access scoped arenas (not that arenas are a
good choice for Clojure, a language that tends to produce a lot of garbage
even in intermediate computations). A type system is just one way of achieving
this isolation, and probably the wrong way to achieve it in Clojure. Plus,
Rust's arena's are either type-specific or rely on reflection.

P.S.

IMO, the coolest thing regarding compiling Clojure is now Truffle/Graal --
HotSpot's next-gen JIT and language-compilation platform -- that will be
available for stock OpenJDK builds in Java 9. It will allow much more
aggressive optimizations for Clojure code. It will also let you write the
Clojure machine-code generation logic in Clojure!

[1]:
[https://github.com/Monnoroch/RustJni/blob/master/tests/main....](https://github.com/Monnoroch/RustJni/blob/master/tests/main.rs)

[2]:
[http://docs.oracle.com/javase/realtime/doc_2.1/release/rtsj-...](http://docs.oracle.com/javase/realtime/doc_2.1/release/rtsj-
docs/javax/realtime/NoHeapRealtimeThread.html)

------
bkeroack
"You're in an industry reaping disproportionate benefit from loose money
policies, leading to a trend-chasing culture of overpaid nerds making web
apps."

This is how software engineers undervalue themselves and damage their own
profession. You will never see doctors or lawyers write a statement like that
about themselves.

The line about "loose money" is laughable to anyone who's ever actually tried
to raise funding.

~~~
nickbauman
Agree. But there is an element of truth to it. And that same element can be
applied to doctors and lawyers. Probably especially lawyers. Doctors and
lawyers don't fundraise for themselves, either. They don't need to.

------
JBiserkov
>Rust is statically typed. The upside is, you will curse it at compile-time
instead of at runtime. The downside is, "exploratory programming" means
exploring how to convince the compiler to let you try an idea.

~~~
alfiedotwtf
For me, yes you will curse _a lot_ at compile time. But the upside is that
stupid, preventable programming errors (which are responsible for a majority
of security vulnerabilites) are stopped at compile time, not at segfault time.

~~~
steveklabnik
We've also found that people generally struggle at first, but at some point,
you internalize the rules, and then they click, and you don't fight all that
much anymore.

~~~
jamwt
Yes, the first few weeks of Rust, I wanted to strangle the borrow checker.
Now, it seldom gets into the way b/c it's a seamless part of my thinking when
I'm writing Rust. I rarely hear from it because I wrote/structured things the
right way the first time.

This has had a huge impact on pushing my productivity in Rust back up toward,
while not Python-ish levels, at least golang-ish levels.

------
scribu
Previously:
[https://news.ycombinator.com/item?id=9250020](https://news.ycombinator.com/item?id=9250020)

I thought it wasn't possible to submit the exact same URL twice.

~~~
dagw
There's a timeout of after which point you can submit a URL again.

------
pygy_
This is a good introduction to Rust, regardless of your Clojure background (I
read a few articles about it when it was trending on HN, but never wrote a
line of Clojure).

------
edem
They call themselves "clojurians" if I recall it right.

~~~
jarcane
Oakes literally wrote the Clojure IDE I use when I'm not on Emacs, so I figure
he earned the right to call himself whatever he likes. ;)

------
lambdaelite
> Rust’s strengths are Clojure’s weaknesses, and vice-versa. Rust isn’t as
> expressive or interoperable, and its concurrency story isn’t as complete.
> That said, it’s much better for performance or safety critical needs, and it
> can be embedded inside other programs or on very limited hardware.

I find this hard to believe. Is anyone actually using Rust for a safety-
critical application?

~~~
jamwt
Depends on what you mean by safety. "People may die" safety, no; but "data may
be lost, existential risk to business" safety, yes. Dropbox is writing a block
storage engine in it.

~~~
lambdaelite
I go by what I thought was the accepted definition, which is a failure or
error presenting a risk for temporary or permanent harm to people. Financial
risk like data loss would fall under mission critical.

------
mej10
I write Clojure every day and love it. I'm interested in learning Rust, but
I'd like to learn it via a project that I couldn't do effectively in Clojure.
Any ideas?

~~~
grayrest
There aren't that many things you can't do in Clojure, the main place Rust has
an edge is in efficiency. I plan on doing an Avro implementation but other
projects (mostly work-related) have a higher priority.

In terms of larger projects, the Servo guys always seem to encourage people
learning Rust to get involved. One of the projects I'm most interested in is
Frank McSherry's timely/differental dataflow repos. I wanted a Rust
implementation of Naiad when I first read the paper so I was quite pleased to
see one of the authors do an implementation, even if it doesn't seem to be a
full time pursuit.

[1] [https://github.com/frankmcsherry/timely-
dataflow/](https://github.com/frankmcsherry/timely-dataflow/)

Most first-rust-projects that are announced on reddit/discourse seem to be
games (Piston), simple web apps, raytracers, or datastructures.

~~~
nickbauman
I _love_ Clojure but it's unsuitable for realtime problems because of GC. For
example, a programmable audio synthesizer. Audio signal processing has to be
realtime or else it can't sync with the music playing. Rust may not be
suitable for that either, but it seems like it's much closer to being able to
do that than Clojure will be anytime soon.

~~~
grayrest
It's not something I've explored or really care to pursue, but I've sat though
at least a half dozen talks on audio synthesis in Clojure including one by
Rich himself. They do signal generation in Clojure that feeds data into
another system that does the real-time portion.

I agree that Clojure isn't suited to hard real time systems and it's not hard
to come up with drawbacks to the above approach but you just happened to pick
an example that a lot of people in the Clojure community seem to care about.

~~~
nickbauman
> feeds data into another system that does the real-time portion

Well of course that's a technique but "another system" isn't Clojure. It does
so happen that a Lisp language is especially fluent for programming audio
processors because of its compiler-like expressions and macros.

