
The Rust Compilation Model Calamity - WTTT
https://pingcap.com/blog/rust-compilation-model-calamity/
======
dwohnitmok
Looking at the TiKV compile time graph, if that's for 2 million lines of real-
world (not synthetically generated) code, that actually seems kind of
reasonable.

~12 min for a full release build? ~5 min for a full dev build? What looks like
several seconds for an incremental build?

I suppose it's good that a (former?) core contributor and large user of Rust
holds the language to such high standards, but this doesn't seem especially
bad.

I'm kind of curious how much faster compiling a large Go codebase is. How
fast, e.g. does the entirety of Kubernetes compile? I'd imagine it's probably
under a minute, but is it several seconds?

~~~
p1necone
As far as I can tell most of the people complaining about Rust having
painfully long compile times are just parroting second hand information and
don't actually know. Kinda like the pervasive "but isn't Java really slow?"
thing that still doesn't seem to have died either.

Edit: yes I know _this_ article was written by someone who knows what they're
talking about (and as they're a steward of the project I understand why
they're calling this out for improvement, and somewhat exaggerating how bad it
is). I was talking about the occasional comments I see on hacker news about
how people aren't even bothering to try rust because compile times are
"unusably" bad. 5 mins for a dev build of _2 million lines_ of code is not
anywhere near reason to not even bother trying the language. No personal
project or learning scratchpad is ever going to come close to 2 million lines.
And it's not worse than many other languages that are used for large projects.

~~~
pjmlp
We do have experience, my C++ projects compile much faster than their rewrite
in Rust.

On common laptops that you buy at the shopping mall, not compiler rigs.

I am not doing nothing special, other than all my third party artifacts are
binary dependencies, not overusing templates, incremental compilation and
incremental linking.

Cargo currently doesn't do binary dependencies, so already here I have to wait
for the world to compile.

And while incremental compiling is already supported, lld still isn't
supported on stable.

Plus I do have experience using other AOT compiled languages with modules
support, since back in the day.

So we know what are talking about, and plenty of people on Rust team are quite
aware of the problem, however not every can be fixed at the same time.

~~~
liopleurodon
> We do have experience, my C++ projects compile much faster than their
> rewrite in Rust

Wow. And C++ projects can be quite slow to compile

~~~
pjmlp
They can, but you have to compile everything from scratch, not use binary
dependencies, disable pre-compiled headers, not having an incremental compiler
and linker available, and be an heavy meta-programming user.

------
WalterBright
I don't know how the Rust compiler is built. However, I am implementing an
Ownership/Borrowing system for the D programming language, and to make it work
requires Data Flow Analysis. DFA is slow. It's normally only used for
optimized builds, which is why optimized builds are slow.

But when DFA is a required semantic feature of the language, it has to be done
for debug builds, too, and so they're going to be slow.

~~~
jez
Can you elaborate on why data flow analysis is slow, necessarily? For example,
does it need whole-program information, or does it do some sort of fixed
point, or is the algorithmic complexity super-linear, or something else?

~~~
WalterBright
It tends to be quadratic, based on the number of variables and the cyclomatic
complexity. The DFA equations cannot be solved directly, but only iteratively
until a solution is reached.

Debug code generation, on the other hand, involves none of that and the
compiler just blasts out code expression by expression. The time goes up
linearly with the number of expressions.

DFA (and the O/B DFA) is done at the function level. It can be done globally,
but I try to keep the compile times reasonable.

~~~
kragen
Can you cache the solution from the last compile and check that it still works
in linear time without introducing nondeterminism?

~~~
WalterBright
Many implementors try to cache the results of DFA and patch it incrementally.
Anecdotal evidence is that they spend endless hours dealing with weird
optimization bugs because they patched it wrong.

I decided to redo the DFA from scratch anytime the AST changed, and have had
pretty reliable optimization as a result.

------
epage
I feel like the breakdown in problems should include real world data to back
it up. Hopefully future articles will do so. In particular, people like to put
blame on the borrow checker when I thought it wasn't all that bad. From my
understanding, the worst offenders are LLVM (because rustc is using it in a
way that they haven't optimized for) and linking (which there are experiments
with using lld which people report huge gains with.

Besides the tone, my main gripe with the article and some discussions I've
seen elsewhere is mixing implementation trade offs with design trade offs. For
example, LLVM and not doing your own optimization passes can be important for
time-to-market. The only reasonable alternative that I can think of without
sacrificing time-to-market is between LLVM or a C backend. Delaying Rust would
have made it irrelevant.

Now for some context for those not as familiar with Rust:

> Stack unwinding — stack unwinding after unrecoverable exceptions traverses
> the callstack backwards and runs cleanup code. It requires lots of compile-
> time book-keeping and code generation.

This is for asserts (panics) and can be toggled with a flag. It isn't inherent
to the language though some older code uses it extensively (like rustc)
because it predates the current language design (from what I've read).

> Tests next to code — Rust encourages tests to reside in the same codebase as
> the code they are testing. With Rust's compilation model, this requires
> compiling and linking that code twice, which is expensive, particularly for
> large crates.

This is in a list of negative impacts of features but people outside of Rust
reading this might not catch the why this is done. Tests inline to your code
have access to your private interfaces. External tests are for integration
testing and build against your library like anyone else would.

~~~
gameswithgo
The worst offenders vary a lot by particular codebase. I had a crate where the
compile times were bad due to trait bounds. I was able to refactor to drop
times from ~2 minutes to ~6seconds! That should'nt have been necessary for me
to refactor though.

Sometimes its llvm, sometimes its rust macros encouraging HUGE functions which
then leads to slow llvm optimizations. Sometimes it's large use of generics.
Sometimes it is the linker. There are various ways to hit slow compile times.

~~~
rhn_mk1
How did you figure out what was responsible for the slowness?

------
jillesvangurp
Compilation time is mostly an issue during the edit, compile, debug cycle and
inside IDEs where things are compiled on almost every key stroke.

I was reading about the rust analyzer recently, which is a new language server
for Rust that is explicitly designed to address compilation latency in IDEs.
This also happened with Java back in the day when IBM developed their own
incremental java compiler for use inside of Eclipse (technically this predates
their IDE; I was using an early version in 1998). It gives that IDE an edge
over things like Intellij in terms of compile latency, which is orders of
magnitudes slower in intellij (measured in seconds instead of ms.). Intellij
does a lot of work to hide the issues through elaborate caching, lots of
things happening asynchronously, etc. They even attempted to integrate the
eclipse compiler. But it's very noticable if you are used to fast feedback on
your code correctness.

Another important aspect that they are trying to address in the Rust Analyzer
that the Eclipse Java compiler also addresses is handling partially correct
code. If you are editing, the end state is of course correctly compiling code.
But in between edits when it doesn't compile is exactly when you need your IDE
to be helpful. Eclipse used to be really good at this and update in real time
on basically every key-press. The red squiggly lines disappearing basically
means "now your code is compiling fine". Intellij works a lot slower and
worse, ends up actively lying quite often with both false positives and
negatives being very common (the dreaded reset caches feature is a top level
menu item for this reason).

So, it's an important problem. Rust is optimized for run time safety and
performance. The same infrastructure that enables that should in principle
also be able to enable a great developer experience when it comes to IDE
friendly features.

[edit] This article what I'm referring to above:
[https://www.infoq.com/news/2020/01/rust-analyser-ide-
support...](https://www.infoq.com/news/2020/01/rust-analyser-ide-support/)
(interview with one of the Rust Analyzer devs)

------
jedisct1
The great thing about Rust is that when you switch to Go, the development
experience feels incredibly fast and great.

Autocompletion in VSCode works, instantaneously, all the time. Errors are
shown in real time. Compilation is fast. Error messages are human readable.
Updating Go or dependencies doesn't break existing code.

------
okareaman
If the code is modularized and put in libraries, how often do you need to
build all 2 million lines of code after making a change. What am I missing?

~~~
reggieband
He does say:

>Per-compilation-unit code-generation — rustc generates machine code each time
it compiles a crate, but it doesn't need to — with most Rust projects being
statically linked, the machine code isn't needed until the final link step.
There may be efficiencies to be achieved by completely separating analysis and
code generation.

It was my first thought as well but I have no knowledge how Rust linking works
so I don't know how reasonable caching module compilation units would be.

~~~
okareaman
Maybe a slow linker is the problem

~~~
Ar-Curunir
The system linked is pretty slow; many folks report massive speed ups by using
LLD

~~~
MaulingMonkey
Replacing BFD with Gold (ELF-only) or LLD can be a _huge_ win.

On the last major production codebase I did this for, linking a single
artifacts went from ~60 seconds (BFD) to ~10 (Gold). Multiply by nearly 100
artifacts (several dozen libraries, several dozen tools, per-library unit
tests, benchmarks, applications, etc. all statically linking some common
code), and some basic incremental "I touched a single .cpp file" builds went
from 30+ minutes to maybe 5 minutes for a single platform x config
combination.

------
Eikon
> TiKV is a relatively large Rust codebase, with 2 million lines of Rust.

I don't understand why so many absolutely wants to inflate this number, as it
doesn't mean anything about the product.

This inflation also leads to making part of the analysis of this article just
wrong.

Running scc in TIKV's "src" folder:

Lines Blanks Comments Code

78442 7259 5476 65707

~~~
azakai
The article says the 2 million includes vendored code. I assume that means all
the dependencies and their dependencies. (So yes, the project itself is just a
small fraction of the total lines, but I'm not sure if the measurement here
was of compiling the dependencies too, or not.)

------
gHosts
Walter Bright has said on several occasions that compilation speed was one of
his design goals for D.

~~~
zaphar
If I'm reaching for Rust it's more than likely because I want it's borrow
checker and pattern matching on enumerated types.

To my knowledge D has neither of those so it's not really in the running here.

~~~
earenndil
A borrow checker is currently in development for d, and pattern matching has
been pseudo-implemented with metaprogramming.

------
toolslive
One of the main reason I got into Ocaml (coming from C++) was that it had
bytecode compilation (and obviously a bytecode interpreter) next to the native
one. This is really fast (5-7 times faster last time I measured) and perfect
during development, when you don't really care about performance.

~~~
pdimitar
I wanted to use OCaml several types but the lack of parallelism and the GIL
have put me off.

Still waiting for Multicore OCaml.

~~~
pjmlp
It supports parallelism the same way UNIX clones did for their first two
decades, which is quite good for plenty of workloads, and then there is Lwt
for concurrency.

And it is safer. :)

In any, case multicore runtime is getting closer,
[https://discuss.ocaml.org/t/multicore-ocaml-
january-2020-upd...](https://discuss.ocaml.org/t/multicore-ocaml-
january-2020-update/5090)

~~~
pdimitar
If we have to mention inter-process communication where every process is
single-threaded then IMO the battle has been lost.

Best IPC is no IPC, wouldn't you agree? In-process function calls can't be
beaten.

Give me a mix of OCaml and Rust with the runtime of Erlang/Elixir and I ain't
learning another programming language until the day I die!

Thank you for the link, I found it really interesting.

~~~
pjmlp
In the times of Spectre, Meltdown and in-process exploits due to threading
issues, everyone doing microservices, I don't see the inter-process
communication route as that bad.

~~~
pdimitar
Sure, pretty good point actually.

I'm just saying that in these same times many problems fall into the category
of embarrassingly parallel and there's no reason to wait 4s for a result that
can easily take 0.5s.

But you're also correct.

------
polskibus
Could the slow compile times be also due to people using laptops with slow
CPUs and drives compared to high-end desktop CPUs and NMVes? We're using
desktop PCs at my work and it made a huge difference in our dev flow (C#, VS)
compared to the Thinkpads we used before.

------
shaggie76
What I'm more interested in their link times; how often do you build
everything from scratch in a day?

------
adamnemecek
You have a monorepo with 2 million lines of code? How long is the compilation
supposed to take?

~~~
disconnected
I don't know how long it is "supposed" to take, but you can compile a Linux
kernel (27.8M lines of code, though a good chunk of it probably isn't going to
be compiled anyway because you don't need it, and another good chunk is
architecture specific) in under 10 minutes on relatively modest (but modern)
hardware.

On the other hand, something like Chromium (25M lines of code) will take about
8 hours, and bring your machine to its knees as it consumes ALL available
resources (granted, last I did this I only had 8GB of RAM, and I was running
my desktop at the time... including Chromium). I don't remember exactly how
long Firefox takes to build, but I remember it was significantly less time
(maybe 3 hours?).

So... it depends? On a lot of things?

(btw, LoC numbers were pulled from the first legitimate looking result I could
find on a quick search... take with a grain of salt... also, compilation times
are a rough approximation based on my observations... that it with a truckload
of salt)

~~~
gpderetta
Linking can consume huge amount of memory, especially for C++ code. For a
large project 8GB might be very low. On our codebase we saw huge differences
between 16GB machines and 24GB machines as the 16GB could not run some linking
steps in parallel without swapping.

Does Chromium build use LTO (I'm pretty sure FF does)? That's also a huge
resource sink a doesn't parallelize as well (a lot of the optimization will be
delayed to linking)

------
carapace
> When I worked daily on the Rust compiler, it was common for me to have at
> least three copies of the repository on the computer, hacking on one while
> all the others were building and testing. I would start building workspace
> 1, switch terminals, remember what's going on over here in workspace 2, hack
> on that for a while, start building in workspace 2, switch terminals, etc.
> Little flow, constant context switching.

And you didn't see the problem!?

When Prof. Wirth was making the Oberon compiler he had a heuristic that any
language feature which made the compiler slower at compiling itself was
reworked or discarded.

------
ansible
Of all the things mentioned in the article that keep Rust compiles from being
fast, the one that sticks out to me is the multi-threaded compilation. That
could result in some dramatic reductions in compile time.

With a lot of the other things that are a part of the design of the Rust
language itself, I'm happy with how they are, even if that slows down
complication somewhat.

------
Ericson2314
I'm glad the comments mostly get it: only incremental build granularity and
parallelism matter. Fully rebuilds I wholely don't care about, and neither
should anyone else: the goal is to never need to do them.

------
scoutt
> Servo is a web browser built in Rust, and Rust was created with the explicit
> purpose of building Servo.

I didn't know this. Is Servo the best example of development/coding
standard/features/implementation for Rust?

------
xiaodai
Is it slow compared to C++?

~~~
antientropic
No. My experience is that Rust compile times are pretty reasonable compared to
typical C++ projects (and it's often an apples-to-oranges comparison: an
initial Cargo build does a lot more, building all your dependencies in the
configuration that you want, rather than using prebuilt binary dependencies as
is common in the C/C++ world). To be honest, I don't really understand the
self-flagellation in the Rust community about compile times.

~~~
nestorD
> I don't really understand the self-flagellation in the Rust community about
> compile times.

I believe the two main factor are the fact that:

\- many Rust users come from languages with a faster writing->running cycle
(there is a surprising number of Python users starting to use Rust)

\- Rust takes prides in its speed, if something is slow it is thus seen as a
failure that should be fixed (even if it has perfectly reasonable reasons to
be slow)

