Hacker News new | past | comments | ask | show | jobs | submit login
The Little Book of Rust Macros (danielkeep.github.io)
205 points by DerekBickerton 5 days ago | hide | past | favorite | 30 comments

Note that this version of the book has not been updated since 2016. The updated version is here: https://veykril.github.io/tlborm/introduction.html

It is definitely also worth mentioning the now-stabilized procedural macros [1] (unless I missed where they are described). proc macros power some of the more complex macro usage in the rust ecosystem (like rocket and actix-webs router macros, any sort of custom derive, etc).

[1]: https://doc.rust-lang.org/reference/procedural-macros.html

Wondering when Rust macros will learn about types.

Lisp macros, of course, know nothing about types, because, in old-school Lisp, there are, lexically, no types. But in Rust, at expansion time, types are usually absolutely nailed down. (...Unless the macro is expanding to a generic definition; but even then, there are named types, you just don't know what they would be bound to).

Or maybe they do understand types, now? I haven't checked lately.

Rust macros run before type inference, and can change type inference in just about every way. They can introduce new types, implement traits for existing types, introduce new variables, and add new type constraints on existing variables.

Rust macros understand types in the sense that they understand which parts of the syntax tree are specifying a type and which are expressions, but I think it's fundamentally impossible for them to understand the actual types of variables.

"Fundamentally impossible" sounds like a challenge. Rust's unbounded type inference imposes quite a large cost that I wonder if the language designers have come to regret. But macros might, in principle, someday be run provisionally, first without inference, and then applying inference to the output, until a fixed point is reached. That would only be if you were willing for Rust compilation to get even slower.

Of course it is easier to talk about this than to design and implement it. Considering embedded and mutually recursive expansions, the Halting Problem rears its head, and there may not be a fixed point, or anyway none in the immediate century. But those would be exceptional cases.

Ok, maybe that claim is a tiny bit too strong, but I'll defend it anyways.

Consider that rust macros are allowed to have side effects and access outside resources, they aren't pure functions. I'd argue that any system that runs them repeatedly in an attempt to find a fixed point is either incorrect (backwards compatibility wise), or an all together new macro system that happens to run in parallel to the current one and call itself the same thing (debatable, I know). Nor can we simulate the proc macro and only let the side effects occur once we reach a fixed point, because the proc macros behavior can depend on the values returned by those external systems.

The halting problem is arguably not a theoretical issue, since the type system is already turing complete (up to arbitrary limits on recursion depth that could exist here to).

I too am pretty confident that there will be no generally type-aware Rust macro system, which is legitimately disappointing.

Still, the types of many named entities will typically be knowable. So, a macro system that can provisionally operate on types when they are known, or even that supports macros that require that the types of certain names be known, seems doable, in principle. It would need a good usage story to get any traction.

This was a great learning resource for macros in the early days of Rust when there was almost no official documentation.

Nowadays anything that's a bit more complex should almost certainly use proc macros instead, which allow much saner implementations than complex, recursive macro shenanigans.

On the other hand, procedural macros are the big guns and as such if you aim them wrong you are going to blow both feet off. A declarative macro is only at worst going to spew nonsense into your post-processed source where it was used - which the Rust tools are quite happy to show you - but a sufficiently bad proc macro could cause unlimited mayhem with no diagnostics available. I don't know if somebody is collecting the worst mistakes they've seen with proc macros, but m-ou-se's nightly-crimes! shows what sort of insanity is possible.

[ https://github.com/m-ou-se/nightly-crimes nightly-crimes! blows away your compiler, running it again in a new environment where it will allow nightly features even though you've got a stable compiler installed... ]

> [ https://github.com/m-ou-se/nightly-crimes nightly-crimes! blows away your compiler, running it again in a new environment where it will allow nightly features even though you've got a stable compiler installed... ]

That macros have access to the entire language including arbitrary IO is the defining feature of proc macros (not without controversy). The insanity here is the Rust compiler team adding the `RUSTC_BOOTSTRAP` env var which is a hack used to bootstrap rustc using a stable compiler despite the nightly features used by the codebase.

All nightly-crimes does is use `std::process::Command` to rerun the compiler with the variable set [1], which tells rustc to throw all concepts of stability out the window.

I haven't been following developments but one of the ideas (even has a PoC iirc) was to distribute and run proc macros as web assembly to improve build times and prevent such shenanigans.

[1] https://github.com/m-ou-se/nightly-crimes/blob/main/yolo-rus...

I don't see RUSTC_BOOTSTRAP as "insanity". Mara could do whatever here. Even if Rust's stable compilers had an internal parameter totally disabling nightly features, Mara's macro could take your compiler apart, identify the parameter, flip it, and re-assemble the compiler then run that. Indeed even if nightly was abolished, and new versions of the compiler simply didn't have these features, Mara's macro could go to github, download the old compiler that did have these features, install and run that.

night_crimes! must be possible unless/ until sandboxing is implementing for proc macros and so the insanity is the same regardless of how it's possible. Which is fine, but like I said, these are big guns, not good to use just because they're tidier than a couple of declarative macros to get your job done.

Absolutely disagree. If it can be done, without sacrificing useability, without proc macros, that's the way it should be done. Proc macros don't play well with IDEs, add compilation time, introduce dependencies, and can be a security risk.

There are things that can be done to improve proc macros, like executing them in a WASM runtime, but currently they should only be used when necessary.

I've used macros for years, decades ago. I don't like them, and I was disappointed to see them so widely employed by Rust.

I've written macros in asm, in TeX, in Lisp, and even in TRAC (a macro based programming language invented by Calvin Moores in the 60's, see [1]). The most impressive macros I've used are found in the fantastic LaTeX graphics package named tikz. The manual for tikz is 391 pages long. This amazing graphics package is implemented in LaTeX and TeX macros.

So if tikz is so great, what's wrong with macros? The tikz package is great in spite of being implemented in macros. Looking over its implementation, I'm stunned that its developers were able to achieve it.

Macros are used because they abstract a mechanism that programmers would like to use, but macros easily become leaky abstractions. They can easily have semantics that depend on or affect other program elements that are not visible to the programmer. Macros can be written using other macros; macros can define new macros. Consequently, even simple macro systems (like the C preprocessor) are capable of doing any kind of a computation [3]. These machinations are not visible to the programmer using the macros so macros make accurate reasoning about the correctness of a program more difficult in the same way that subroutines having unrestricted access to global variables make correctness harder to (informally) verify.

Abstraction has proven to be an essential means for constructing programs. No one wants to program in an environment where goto's in a flat namespace replace all functions. I feel like macros are a flawed mechanism for abstraction; I would rather see the language syntax expanded to accommodate the desired features than to have them implemented via macros.

As impressive as the capabilities of macros are [4], wouldn't it be better to rely on functions as the primary abstraction mechanism used by the language's programmers.

Am I wrong?

[1] https://dl.acm.org/doi/10.1145/365230.365270

[2] http://www.bu.edu/math/files/2013/08/tikzpgfmanual.pdf

[3] https://stackoverflow.com/questions/3136686/is-the-c99-prepr...

[4] https://letoverlambda.com

Regular Rust macros are hygienic, unlike macros in many (but definitely not all) other languages. This means that a macro expands in place in the AST, and can only operate on its inputs. The C preprocessor, by comparison, operates on text and the expanded macro can trivially interact with syntactic elements around it once expanded.

Procedural macros can run arbitrary rust code and can therefore do anything, but in general they don't -- and if you work within the interface you're given then the result will be hygienic.

There's a sense in which macro hygiene is analogous to Rust's safety guarantees -- the compiler will normally guarantee the abstraction, but there's an escape hatch where we rely on the person implementing the code that uses unsafe or implementing a procedural macro to not break the abstraction for their callers. A leaky abstraction is considered to be a bug.

By co-incidence, I started reading the updated version of this book today! Fantastic resource.

Question to experienced Rust people: What are some cool use-cases that you've seen for Macros?

Diesel is the leading ORM for Rust and they allow you to build SQL queries in a type safe way, as well as, making sure the struct you deserialize to matches the output of the query (ie, type safe deserialization).

I've used ActiveRecord and Django's ORM, and neither provide anything like this.

The closest I've seen is SQLModel, a library on top of SQLAlchemy built by the author of FastAPI.

However, last I checked, the kinds of queries that Diesel allows are more restricted than that of Django or ActiveRecord. Of course , you can always write custom SQL for things like that when using Diesel.

I believe also that Diesel is either maintained by or was created by the same peeps behind ActiveRecord. I also read somewhere that a running joke among them is that they learnt what not to do in writing an ORM through ActiveRecord.

A question I have for y'all is: can anyone make a comparison between Diesel and TypeORM?

Serde is really the killer app for rust macros IMO

Serde is honestly just so amazing. It amazes a lot of people when I demo rust and show how trivial it is to read and write our existing configuration and data files

I assume you're asking about authoring one's own `macro_rules!`? I find them handy when writing tests (especially integration tests) to create sub-modules that all function the same way, just with different input-output pairs. This is better than using functions because each module can contain multiple functions to test. So instead of `test1a(input1, output1); test1b(input1, output1)`, I have a macro `test_input_output!(test1, input1, output1)` which generates `mod test1` containing the individual tests.

To me, the coolest is sqlx. It uses macros to do static type checking on plain SQL statements. It verifies types of bound parameters as well as results. It's async, which plays nicely with the leading web frameworks, etc.

[0] https://docs.rs/sqlx/0.5.9/sqlx/macro.query_as.html

This is probably a relatively boring use case -- the C preprocessor could do basically the same thing -- but I've used macros to obviate some repetitive code in API bindings and FFIs.

I'm using proc macros to generate a parser implementation from a context-free grammar, I find it pretty neat.

Is this open source anywhere? I'd love to take a look at the code. A compiler is on my short-list for my next project to use to learn rust.

Not yet

If you have used Rust in any real world capacity on actual projects you know that there are no "Little Books" when its comes to using Rust. An overly complex and unproductive language as much as C++.

Absolutely not my experience. My experience is there is an initial large learning curve (2-3 weeks) and then you can be vastly more productive than any dynamic language, with better safety, a better library ecosystem, and better tooling.

Both are overly broad generalization. If you are working on a backend for a web app I would certainly not recommend Rust and it would be a big loss of productivity compared to Python or JavaScript, or even Go.

For a game it could fit in some part but you would be very limited by the ecosystem so I would also not recommend except if you really know what you are doing and have significant resources.

For low level system programing it is pretty nice, and maybe the best alternative right now.

For a compiler I would say it depends if performance is the main priority, in that case yes, otherwise no.

There are plenty of other cases of course and for each the answer would be different.

It is not really a general purpose language that you could use without worry for everything like Python, at least not yet.

Using libraries is so much easier in rust compared to many other libraries due to rustdoc. I can trust the type signatures to show me the actual usage, I can trust the code snippets to not be outdated, and above all this is consistent for all code in the entire ecosystem.

Almost always when I use python libraries I have to get used to a new documentation format, learn how to navigate it and so on. And then it has to be detailed enough to make up for the lack of type signatures. I cannot tell you how often I have to skim a significant portion of my dependencies' source just to use them. (Though there are counterexamples, stdlib and numpy in particular actually have okay docs)

I would absolutely do web backends in Rust, with Go as the backup choice. Python doesn’t feature in my top 10, and nor does JavaScript without TypeScript to make it even moderately reasonable.

Wouldn't you say this about Go before Rust?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact