

How to Write a Rust Syntax Extension - neverminder
http://brodoyouevencode.com/posts/how-to-write-a-rust-syntax-extension/

======
ajross
I can't help thinking that this is just the wrong way to do this.

If I have a subtle metaprogramming problem that I'm going to solve with the C
preprocessor, I can code it and document it right there in the same source
where it's used. Because that's the natural place for it. The metacode is the
fundamental thing I'm writing, and it belongs in the "code" files for the rest
of the project.

These Rust things are implemented as modules. I need to keep them separate. I
need to build them separately. That breaks discoverability really badly. If a
future maintainer needs to figure out what my strange macros do, they just
look at them. If they get confused by my strange syntax extensions, they need
to figure out where some other crate is coming from.

Regardless of other concerns about the mechanics, I think this is going to put
a really serious damper on this kind of code.

Quite honestly, I'm looking at this and seriously thinking that more
pedestrian options (python scripts to emit or preprocess rust code, for
example) might be simpler and cleaner in practice.

~~~
mbrubeck
Rust also has macros, which can easily be defined next to their use:
[http://doc.rust-lang.org/book/macros.html](http://doc.rust-
lang.org/book/macros.html)

Syntax extensions are more like plugins for the Rust compiler.

~~~
ajross
I know, but Rust macros seem to be fairly simple "expansion of a syntax tree"
things, not intended as a programming environment. This is (or at least
appears to be) the Rust answer for things like elaborate C macros or C++
template metaprogramming. There's a place for that kind of code in the world,
even though it's needed rarely.

Thinking about this as a "compiler plugin" is exactly what I think I'm getting
at: if I need to implement some sort of DSL feature for my code, _I want to do
it as part of my code_ and not have to deploy it as a plugin to the compiler.

~~~
pcwalton
> This is (or at least appears to be) the Rust answer for things like
> elaborate C macros or C++ template metaprogramming.

Generics are Rust's answer to most template metaprogramming.

> if I need to implement some sort of DSL feature for my code, I want to do it
> as part of my code and not have to deploy it as a plugin to the compiler.

The problem with that is that, in a cross-compilation scenario, the
environment that the compiler runs in may be completely different from the
environment that your code is deployed in. Different symbols may be available,
some libraries may or may not be there or work differently, etc. So you really
need it to be separate modules.

~~~
ajross
That seems kind of non-responsive. The fact that there are technical
limitations with Rust's mechanism doesn't invalidate the fact that the
existing solutions work fine when implemented with a cross compiler and yours
don't. It's sort of a special case of my point: maybe you did it wrong.

( _Edit, for clarity: the C preprocessor and C++ template languages are well-
specified interpreted languages executed at compile time. Rust 's attempt here
is poorly specified and coupled closely to the runtime environment of the
compiler binary. And that choice has leaked into the API presented to the
programmer. And that's bad._)

And regarding templates/generics: I'm not talking about type calculus stuff,
but more like the kind of code you see in Boost's lambda or serialization
libraries (which to be fair I think are terrible libraries personally -- IMHO
this kind of code doesn't belong in libraries at all, which is my point). I
don't think Rust generics (to their credit) can do that.

~~~
pcwalton
> That seems kind of non-responsive. The fact that there are technical
> limitations with Rust's mechanism doesn't invalidate the fact that the
> existing solutions work fine when implemented with a cross compiler and
> yours don't. It's sort of a special case of my point: maybe you did it
> wrong.

Two of your examples have been template metaprogramming and cpp, both of which
are very limited programming environments (technically Turing complete, but
both are Turing tarpits). When you actually want a full programming
environment, you always hit the cross-compilation issue. It's just fundamental
to the problem. But the advantages of a having a full programming environment
available, _when you need that power_ , outweigh the downsides of having to
write in a separate module.

The other example you gave is writing a Python script, which has all the same
problems of writing in a separate module, except now you're writing in a
different _language_ in addition to writing in a separate module.

Sometimes you don't need the power of a real programming environment, of
course. That's why the macro system, which does not expose a full programming
environment, exists in Rust. _But C++ templates and (especially!) cpp weren 't
designed as programming environments either._ They're pretty terrible
programming environments. If you want that kind of thing, use the macro
system. It's technically Turing complete too in just the same way as templates
are. But we _also_ expose a full programming environment, if you want it.

I'm mystified by how offering a feature that extends the existing state of the
art _in addition_ to offering a feature that reflects the state of the art—the
macro system—could possibly be somehow "doing it wrong".

~~~
ajross
> I'm mystified by how offering a feature that extends the existing state of
> the art in addition to offering a feature that reflects the state of the
> art—the macro system—could possibly be somehow "doing it wrong".

That's a total strawman (seriously: every time I see you deal with a criticism
you pull out this flamage toolkit; please stop). All that stuff in the first
half of your sentence? That's the stuff you're doing _right_.

What you're doing wrong is that I can't write that inline in the code and need
to write it as essentially an extension library for the compiler.

I don't think it's such a strange request that my DSL implementation look like
it does in C or Scheme, e.g. "[Define some quirky DSL hackery] [Implement a
bunch of code using it]". I'm likewise mystified why you think that's not a
useful requirement or a shortcoming (even a minor one -- you're literally
refusing to engage me on this) of your design.

The example code in the linked article is simply _ugly_. Ugly APIs are bad,
even (especially) for subtle features.

~~~
oldmanjay
I'm having a hard time understanding your objection. The nearest I can parse
from all of your posts is "you aren't doing things the way I like therefore
you are wrong" which isn't a very compelling argument to me.

~~~
ajross
Uh... what other definition is there for thinking something is wrong? I've
written DSLs, and code generation hacks, and used the preprocessor, and lisp
macros extensively to do it (not templates so much, to be fair). I know the
problem area, I know what I like, and I don't like this.

It seems that your position is "well, you won't get it so shut up about it".
That seems rather uncompelling too. :)

------
JoshTriplett
Great to see documentation on syntax extensions; hopefully this will end up in
the official Rust book at some point.

Also, this domain makes it _really_ hard to take the content seriously...

~~~
Argorak
Note that syntax extensions are behind a feature gate, that means they are an
experimental feature usable on nightlies, but not on released versions of
Rust.

~~~
steveklabnik
Yup, this is very much why I have put zero effort into documenting them: they
deeply rely on compiler internals, and I have enough other work to do without
helping create de-facto standardization, like what happened with macros...

That said, still glad to see posts like this.

------
pron
> Syntax extensions are a cool feature (and Rust is the only language I know
> of that supports something like this) that allows you to modify the AST at
> compile time, so you can generate your own code or modify existing code.

Java has had this feature for about 10 years now. The standardized API
(required by all conformant compilers) defines only _reading_ a _partial_ AST
(just declaration) and generating new source files, but javac supports full
AST transformations.

Two famous Java projects making use of javac AST transformations are the good-
old Lombok project and Checker, Java 8's new pluggable type systems.

~~~
ajross
Lisp has had this feature for... never mind.

