
ClojureC, a compiler for Clojure that targets C as a backend - terhechte
https://github.com/schani/clojurec
======
mullr
This is neat.

What worries me about this and similar efforts (like
<https://github.com/takeoutweight/clojure-scheme>) is that clojure's standard
library design assumes that the underlying runtime will be do some kind of
polymorphic method inlining.

For example: the sequence library is all defined in terms of ISeq, which
basically requires a "first" and "rest" to be defined for the data structure
in question. These are polymorphic: there are different implementations of
these for different data structures (list, vector, map, etc). So a dispatch
step is required to choose the right one. In clojure-jvm, this is implemented
using a java interface; this means the jvm will inline calls to said methods
when they're being used in a tight loop. And if you use the standard library,
calls to 'first' and 'rest' are going to be inside nearly all of your inner
loops.

Compare this to a normal lisp or scheme: 'first' and 'rest' (or 'car' and
'cdr', whatever) are monomorphic. They only work on the linked-list data
structure. So compiling these directly down to C functions makes perfect sense
and incurs no performance penalty.

So in summary: clojure assumes theres a really smart JIT which is helping
things along. This means it's not as suitable for alternate compilation
targets as you might want it to be.

I wonder if there's something clever you could do here. Vtables could be
reordered based on expected usage, certainly. Clojure can already do some
measure of type inference, so this could be used for AOT inlining when it's
available. Even if it's not, perhaps several versions of a call could be
speculatively generated based on what the compiler _does_ know already. The
normal polymorphic inline caching technique could perhaps be abused to apply
here. But it's hard to see how any of this can work in absence of a profile or
heavy hinting.

(not a compiler writer, just interested in the problem)

~~~
vidarh
You can do polymorphic method caching/inlining with a hybrid ahead of time /
JIT compiler targeting C _reasonably_ easily. The code fragments required for
caching at least will be small, and code to generate them at runtime is not a
big deal. I'm playing with a Ruby compiler, and Ruby badly needs these type of
optimizations to get fast, so I've spent a fair amount of time looking at it.

For a fair amount of cases you can do static analysis to get good guesses at
likely types, even for cases where you can't be sure. E.g. speculatively even
looking near call sites by method _name_ to see if you can guess the type of
objects that will get passed in looks to get you a reasonable chance at
guessing at the top contenders to let you speculatively generate inlined
versions without creating too much junk. But to get the most performance out
of this you're likely to need to be prepared to do some very basic JIT.

~~~
mullr
>E.g. speculatively even looking near call sites by method name

That is devious and fantastic.

> But to get the most performance out of this you're likely to need to be
> prepared to do some very basic JIT.

Yeah. But the attractive targets here are places where you can't have a JIT:
embedded systems and iOS.

~~~
vidarh
When you spend a few years speculating about what it would take to efficiently
compile Ruby as statically as possible (I love Ruby, but I hate moving parts),
devious becomes second nature...

The idea of speculatively looking at method names comes from testing that to
create vtables ahead of time for Ruby classes, to avoid hash tables in the
common-case.

As it turns out, most method on most Ruby classes are the ones inherited from
Object or other standard classes, and the number of classes is usually fairly
constrained, so again speculatively looking at method names in the compile-
time available source and allocating sparse vtables for the most common names
results in relatively little waste.

And it reduces typical method lookup to a vtable lookup for common methods,
with expensive method dispatch becoming much more rare. There's the tradeoff
between theoretical horrible blowup in vtable waste from apps dynamically
adding tons of methods and tons of classes, with a unique vtable slot required
for each method name across all classes, vs. falling back to doing hash-table
lookups all the way up the inheritance chain for "unusual" method names ones
you reach certain thresholds for waste.

You do incur the cost of propagating vtable changes down the inheritance tree
when methods are dynamically redefined in other places than leaves, but it is
fairly rare to see apps where this happens at a very high rate, and the number
of subclasses usually fairly small, so it is likely to be quite cheap. Doing
it that way is something I first saw in a technical report by (now) prof.
Michael Franz from '93 or '94 on "Protocol Extension" for Oberon.

You can probably also get some decent gains by adding heuristics to give
preference to names that appears to be used in loops when picking names for
the vtables to reduce the need of any JIT'ing.

~~~
lobster_johnson
Out of interest, are you actually working on something like this for MRI? As
you say, Ruby is in desperate need of optimization.

~~~
vidarh
No, I've been off-and-on, toying with writing a "as static as possible" Ruby
compiler (see <http://www.hokstad.com/compiler>) - it's been about two years
since I last posted an update, but I have one new part complete and another
one mostly complete. Just holding off posting for a bit longer because I want
to have a bit of a buffer (3-4 complete parts) before I get peoples hope of
regular posts up again...

What is there uses vtable's exclusively - I effective punted on the slow path
(and so on adding methods at runtime) completely, but keep track of how much
of the vtable allocations is wasted space. If/when I get there, the goal is to
use various mechanisms like this to determine when to fall back on a slow
path, and couple both with polymorphic inline caching when suitable.

EDIT:

I don't see MRI as very interesting to work on, largely because interpreters
aren't much fun, and ironically given the amount of time I spend using Ruby, I
prefer compilers to be as static as possible. I also prefer my compilers to be
bootstrapped in their target language. Hence my "ideal" Ruby compiler would be
written in pure Ruby, do a ton of static analysis, with minimal fallback to
JIT when users user features that are too dynamic to analyse fully ahead of
time

E.g. there's a ton of annoying uses of eval() in Ruby code where a more
complete meta-programming API would make it trivial for a compiler to do full
ahead of time static analysis, so one big thing an AOT Ruby compiler really
need to do is to provide a library of compiler specific meta-programming
facilities with a fallback that uses eval() as needed, and either convince
people to use it, or provide monkey-patches for a number of popular projects.
Some of these uses don't even need eval() in the first places, but uses it
just as a quick shortcut because it's simpler...

Just to make clear, I'm not sure when or even _if_ my compiler project will
ever get to a state where it's even remotely _useable_ to compile Ruby. I
started it out without even having decided to compiler Ruby, mostly to write
about various parts of the process of writing a compiler that I find
interesting. I find compiling Ruby as incredibly fascinating from a
theoretical point of view because of the complexity involved, but
unfortunately working on it takes a lot more time and effort than thinking
about the problems.

~~~
lobster_johnson
I see. Sounds like an interesting project.

Sure, MRI is boring, but it's the best implementation right now (though some
may argue that JRuby is better), and it's in desperate need of VM innovations.

Any new compiler/VM starting from scratch will be years away from being
available for use in production environments. By the time it's finished, we'll
all be using Go. (Sigh.)

~~~
vidarh
I think we need both. MRI can keep getting better, as can JRuby (which is an
amazing feat, but to me, running on top of the JVM makes it a non-starter) or
Rubinius, but they're fundamentally side-stepping the really hard problems.

E.g. nothing will stop MRI from having to interpret thousands of lines of code
each time because it can't draw a line between runtime and compile time, while
for an ahead of time compiler for Ruby, finding a pragmatic line between what
needs to be executed at runtime vs. compile time is essential (consider for
example the tendency to do stuff like getting the list of files in a directory
and require all of them in turn).

~~~
lobster_johnson
We do need both. But some of the things you mention (like constructing
vtables) could be applied to MRI's VM model without writing a compiler from
scratch. Frankly, I would much rather have large performance improvements now
than in 2-3 years.

(I agree about JRuby. I also wonder why Rubinius, which showed so much promise
at the beginning, has stagnated. Is it simply the lack of developers?)

~~~
vidarh
I agree the performance increase would be great, but I think it needs to come
gradually to MRI. E.g. trying to do anything fancy with the old AST-based
interpreter would've been pretty pointless. After YARV, it is probably
starting to get more attractive, but at the same time they've added method
caching which gives a decent amount of the benefits. A vtable will still get
faster, but it might not be the most immediately expedient way of speeding
things up vs. e.g. Sasadas latest project of adding a generational gc.

Regarding Rubinius, writing compilers for dynamic languages is hard. Most
textbooks you'll find cover techniques most suitable for statically typed
languages (the best resource I know for starting to catch up on compiling
dynamic languages is actually the Self papers). So you need more than an
unusual level of interest in writing compilers to be likely to try to tackle a
language like Ruby which is tricky even for dynamic languages (e.g. my
favorite pet problem to meditate on: What constitutes 'compile time' vs.
'runtime' for ahead of time compiled Ruby?), and even more to actually
persevere until you start getting proper results where you can get decent
results in _days_ with a simpler language.

It's made worse because of Ruby's _horrendous_ grammar. And I mean that from a
compiler writers perspective - as a developer I love to _use_ Ruby to a large
extent because the complexities of the grammar means it reads and writes
better 95% of the time. But MRI's bison based parser was 6k-7k lines with ugly
parser/lexer interplay last time I checked... There are full compilers
substantially smaller than that for other languages...

To me, these complexities are part of what makes it fascinating. I firmly
believe you can parse Ruby fully with a much, much simpler parser for example.
A lot of the ugliness can be abstracted away, and C parser code is rarely good
examples of succint code.

I _did_ start playing with MRI years ago, specifically the parser, actually,
and started chopping out redundant pieces, but got frustrated and bored with
it. That's part of the problem - it's one thing to play around with a toy
compiler like I've done, and another entirely to put in the effort to push a
major change to MRI through to production quality given the number of years of
accumulated history encapsulated in it. Doing the latter as a hobby is a
daunting task.

~~~
lobster_johnson
Just a note on Rubinius: The PyPy guys seem to have done pretty well at this.
I don't know how similar they are to Rubinius; PyPy reduces to a RPython as an
initial step, whereas I believe Rubinius compiles to LLVM's LI.

------
saosebastiao
So if I wanted to write CLI applications in Clojure, is this my best bet?
Cause the JVM is about the least suitable platform I have ever worked with for
CLI apps...which is most of what I do. I'm constantly in this pickle of
wanting to use Clojure but defaulting to Ruby because the JVM is so terrible
at it.

~~~
bbq
If the startup slowness is your problem, look into Nailgun:
<http://www.martiansoftware.com/nailgun/>

If that's not your problem with the JVM, what is?

~~~
Rayne
Nailgun is not a good solution to the problem. If the JVM startup speed is a
problem, your best option is to use something not on the JVM and not use hacks
that make it feel faster.

~~~
bbq
That's not true. Yes, it's worth considering outside of the JVM if being on
the JVM means your application isn't interactive enough (startup speed doesn't
matter for long running, fire & forget tasks).

At the same time, splitting your application into a client/server architecture
is not a hack but an engineering decision. There are times when this decision
is natural e.g. Music Player Daemon (MPD)[1]. For most CLI applications,
there's no clear benefit (but the general approach has no clear downside
either - the code overhead of this approach can be brought very low).

Certainly, in a production application you would want to secure the messaging
channel (Nailgun doesn't).

[1] A music playing server: <http://www.musicpd.org/>. Some of the clients
happen to be command line:
<http://mpd.wikia.com/wiki/Clients#Command_Line_Clients>

------
jgalt212
As someone who doesn't use Clojure, but watches it fairly closely, I'd say the
least appealing part of Clojure is its reliance on the JVM.

As such, I'd say efforts such as these are greatly welcomed.

~~~
pjmlp
> As someone who doesn't use Clojure, but watches it fairly closely, I'd say
> the least appealing part of Clojure is its reliance on the JVM.

Like it or not, this is what made Clojure successful in the enterprise, at
least when compared against other Lisps.

~~~
lispm
Is that based on numbers or a guess? How many Clojure applications are there
in comparison to Lisp or Scheme?

~~~
pjmlp
Based on guess.

I see lots of Java shops now having Clojure code and talking about it at JUGs
and how it enables their business.

You just need to have a look at InfoQ, Skills That Matter, Devoxx or Jax for
ongoing talks.

------
billsix
For those interested in Lisp implementations which compile to C, providing
cross-platform benefits and a nice FFI, gambit-c and chicken are performant,
mature implementations of the Scheme programming language.

------
densh
Why C but not llvm? Structured code generation is always better than string-
based one.

~~~
pat_punnu
Can you show a proof for that claim?

I believe it to be nonsense.

~~~
bratsche
I'm not sure if this is what the previous poster was talking about, but clang
has some APIs that let you get access to the AST pretty easily. It's been a
really long time since I've looked at it, and at the time I don't think it was
exposed as a library API for general consumption. But for example:
[https://github.com/bratsche/clang/blob/gtkrewriter/tools/cla...](https://github.com/bratsche/clang/blob/gtkrewriter/tools/clang-
cc/RewriteGtk.cpp)

~~~
pat_punnu
What is his definition of better? It sounds like he thinks it's entirely
objective, so he should be able to express it clearly and logically.

Does he think that it's technically more powerful? Again he should be able to
prove that if that's the case.

Otherwise he's just giving a shitty opinion, and should say that.

I think the claim is nonsense because with inline assembler there is nothing
that you cannot express in C that you can with LLVM. So the decision between
the two is opinion.

~~~
coldtea
> _Does he think that it's technically more powerful? Again he should be able
> to prove that if that's the case. Otherwise he's just giving a shitty
> opinion, and should say that._

It's not like it's some controversial opinion what he said -- it's both self
evident and common place. It's you who offers the more controversial opinion
(and in a rude way, to top).

> _I think the claim is nonsense because with inline assembler there is
> nothing that you cannot express in C that you can with LLVM. So the decision
> between the two is opinion_.

It's not about "expression", and nobody argued that you can express more in
LLVM.

This is missing the point by miles!

It's about having more structure and less of an ad-hoc pipeline, which helps
with better tooling, error prevention, etc.

(Not only what you wrote is wrong, but even if the original argument was about
expression, your opinion would still be wrong. Two things offering equivalent
expressive power, does not mean that they are just as good to use in practice
at all. Might as well ask "why invent new languages, when assembly can express
everything").

The only benefit to using C for something like this is portability, which is
something else altogether.

~~~
vidarh
> It's not like it's some controversial opinion what he said -- it's both self
> evident and common place.

As someone who has written more than one compiler, I don't see how it is self-
evident at all. It's also not at all that common-place compared to generating
C or asm output textually.

> It's about having more structure and less of an ad-hoc pipeline, which helps
> with better tooling, error prevention, etc.

Those provide some benefits, sure. At the cost of massive amounts of
complexity in the case of LLVM.

> The only benefit to using C for something like this is portability, which is
> something else altogether.

Now it is you who are wrong. Other people have already pointed out, for
example, that C provides an easy-to-read intermediate format, and is simple to
generate, as other benefits. Not having to deal with a massive C++ codebase is
another.

You may disagree that these other benefits are worth it, but for me at least
they are (just taking a break from a compiler that generates textual _asm_
because I find even that preferable to dealing with LLVM).

~~~
coldtea
Sure, I agree about this: "C provides an easy-to-read intermediate format, and
is simple to generate, as other benefits. Not having to deal with a massive
C++ codebase is another.".

So portability and less dependencies, plus easier.

------
akkartik
_"Before you can run anything make sure you have GLib 2 and the Boehm-Demers-
Weiser garbage collector installed."_

Wow, that's a pretty skimpy list of dependencies. But..

 _"Make sure you're using Leiningen 2."_

..argh, installing that on ubuntu that requires 110 packages. All that just
for a build system?

~~~
uvtc
> ..argh, installing that on ubuntu that requires 110 packages. All that just
> for a build system?

Don't install lein using the OS packaging system (apt, rpm, yum, etc.).

Instead, just grab the `lein` script (linked to from <http://leiningen.org/>
), put it into your ~/bin, set it executable, and you're all set.

~~~
michaelochurch
This. It's a lot easier to do it that way.

------
timbaldridge
sadly, this doesn't seem to support any sort of multithreading. Even something
as simple as swap! isn't thread-safe in this implementation. So that kills one
of the main reasons to use Clojure in the first place.

~~~
swannodette
Lack of multithreading seems like a result of the implementation being heavily
based on ClojureScript ;) (it's actually pretty cool to see how reusable
core.cljs is IMO). I imagine ClojureC will have its uses like ClojureScript
does when the JVM is not an option.

------
iso8859-1
How does this compare to Chicken Scheme?

------
pjmlp
This is quite nice, on the other hand, one could just use one of the many Lisp
compilers existing since years.

