
A Few More Reasons Rust Compiles Slowly - Caitin_Chen
https://pingcap.com/blog/reasons-rust-compiles-slowly
======
ajtjp
As a relative neophyte in Rust (have gone through about half the chapters in
the Rust Book), I recently deployed a small Rust server in DigitalOcean, and
was surprised by the compilation speed. The server's code was about 27 KB in
size, producing a binary of about 160 KB in size. But it had 92 dependencies,
including transitive dependencies, and took 5 minutes and 38 seconds to build.
Which was quite impressive relative to the size of the code, even allowing for
the not-super-fast CPU.

While I was watching its output, I realized that in Rust, all the dependencies
are compiled; I'm used to Java+Maven or JavaScript+NPM where compiled
dependencies are used instead, and that tends to be pretty quick (provided
your network pipe is wide enough). I'd be curious to learn why Cargo re-
compiles from scratch instead of offering pre-compiled binaries as well. I
guess part of it is related to different target platforms, but it seems like
if the top 5 platforms were targeted and had compiled resources available, you
could reduce compile times for those platforms by a significant amount.

On the other hand, the error messages I ran into along the way were quite good
at pointing me in the right direction about how to fix them, which saved more
time than the extra compile time cost, relative to the Python-based
alternative I had been trying to set up before that.

~~~
steveklabnik
It's not just target platforms, it's also build flags; you could do the
default for debug and release, but set any sort of custom option and you're
back to square 1.

We would also need to pay for the cost and such of building, hosting, and
distributing all of that...

There are other middle grounds too. There's certainly interest, it's just not
trivial. If it was, we'd do it!

~~~
matthewaveryusa
Just out of curiosity, How does rust prevent the Gentoo issue of: you can
compile with custom flags, but most projects have only been tested with
specific flags and if you change them they will break the build, so really, no
customer flags. It always starts off with custom flags and ends up ossifying
to the default-only flags.

~~~
ohazi
Most of the flags tell the Rust compiler what strategies to use when producing
machine code. Actual Rust code rarely looks at these flags, so correct
interpretation of the program generally isn't affected by them.

Contrast this with C and C++, where it's common to #ifdef in an entirely
different program depending on the flags.

It's certainly _possible_ to do something like this in Rust, but in practice
it's rare.

------
ChrisSD
Rust supporter: Build times are comparable to C++.

Rust critic: Build times are comparable to C++!

~~~
uluyol
I wonder whether C++ modules will give C++20 the edge here.

~~~
nindalf
I asked someone who worked on the C++ modules proposal and they said it might
both help and hinder compile times.

On one hand, compile time would be lowered because compilation units are
larger and less can be done in parallel. But in future, incremental builds
probably become quicker because it’s easier to tell what’s actually changed
and what that affects.

------
qppo
I have a rather large side project in Rust using cargo workspaces and
compilation times do not bother me in the slightest, because I'm running
incremental builds for everything.

On the other hand I tried introducing Rust for a small part of a larger
ecosystem and the cold compile times were so bad we rewrote the functionality
in C. It shaved _minutes_ off our CI build times, which costs actual money.

~~~
MaxBarraclough
How do the Mozilla folks cope with these build-time issues?

~~~
zelly
because, it's _checks notes_

    
    
      % curl -Lo mozilla.zip https://hg.mozilla.org/mozilla-central/archive/tip.zip
      % unzip mozilla.zip && cd mozilla-central-*
      % find . -type f \( -iname '*.cpp' -o -iname '*.cc' -o -iname '*.cxx' -o -iname '*.c' -o -iname '*.h' -o -iname '*.hpp' -o -iname '*.hxx' -o -iname '*.hh' \) | wc -l
      32347
      % find . -type f -name '*.rs' | wc -l
      7669
      % find . -type f -iname '*.js' | wc -l
      70101
    

...still mostly C and C++ and more JavaScript than Rust.

~~~
orf
Does that include any of the dependencies? [1]

1\. [https://hg.mozilla.org/mozilla-
central/file/tip/Cargo.toml](https://hg.mozilla.org/mozilla-
central/file/tip/Cargo.toml)

~~~
gpm
Cargo.lock is probably the more useful file for listing dependencies

[https://hg.mozilla.org/mozilla-
central/file/tip/Cargo.lock](https://hg.mozilla.org/mozilla-
central/file/tip/Cargo.lock)

------
pjmlp
> I imagine the prior art is extensive, but a notable innovative responsive
> compiler is the Roselyn .NET compiler; and the concept of responsive
> compilers has recently been advanced significantly with the adoption of the
> Language Server Protocol. Both are Microsoft projects.

Some examples of prior art in the context of strongly typed compiled languages
with compilers designed for interactive development from the get go.

Mesa/Cedar environment at Xerox PARC, and how Oberon and its descendants used
to be integrated with the OS (Native Oberon and others)

Energize C++ and Visual Age for C++ version 4, although quite resource
intensive for their time.

Eiffel, with its MELT VM on Eiffel Studio for development and AOT compilation
system C and C++ compilers for deployment.

C++ Builder and Delphi environments, although still batch, provide a similar
workflow.

~~~
barrkel
Delphi, when providing support to the editor, compiles the whole file
containing the cursor but completely skips all codegen and in fact all
function bodies (i.e. type checking, expression analysis etc.) that don't
contain the cursor. The cursor is represented as a fake token, and the parser
longjmps out with context when it's discovered; it resets the lexer context if
it finds the cursor in the body of a function it's trying to skip.

It doesn't need to compile dependencies because the public parts of units are
a serialized symbol table with full definitions, rather than simple object
files.

~~~
pjmlp
Thanks for the overview, I need to start collecting your comments. :)

------
omn1
Shameless plug from a fellow rustacean here. If anyone is looking for ways to
improve compile times, I recently wrote an article with some tips:
[https://endler.dev/2020/rust-compile-times](https://endler.dev/2020/rust-
compile-times)

------
rubyn00bie
Obviously just a band-aid for the symptom, and probably known by most, but
FWIW sccache has been pretty freakin' great for me at keeping build times
manageable:
[https://github.com/mozilla/sccache](https://github.com/mozilla/sccache)

And, LOL, just now finally read the readme, didn't even know I could archive
the the cache over the network.... #foreverN00b that's gonna be awesome.

------
epage
For build scripts, I've switched to avoiding build-time codegen and instead
doing development-time code-gen, commit the results in, and have CI verify
that the committed results and generator are not out of sync.

This saves me building build-script dependencies, building build-script, and
running running the build-script. In addition, by default, these are built and
run in the same mode (debug/release) as the target binary, making build-
scripts even slower when doing release builds.

Wrote [https://github.com/crate-ci/codegenrs](https://github.com/crate-
ci/codegenrs) to help with this.

~~~
swsieber
I like this approach. While I don't use it for any Rust programs I'm working
on, I do use it for certain java reflection based things.

~~~
epage
I'm working on trying to get my company to switch to this approach for our
python monorepo.

It frustrates me that I've never seen a good code-gen story for python \- Some
do runtime code-gen which is hard to learn from, debug, and verify \- Some do
install-time code-gen, slowing down install and and making it hard to inspect
and verify \- Some check codegen in but without any story to ensure it does
not drift from the generator

------
lukevp
What is “slow”? I mostly do JS and .NET, and I just started working through
the rust hello world. I was shocked that I had an exe in around 0.2s. I
haven’t worked on a large project so maybe this doesn’t scale, but of the 3
(at least for hello world which is fairly trivial) rust felt super fast. I saw
mention of Roslyn so perhaps the issue is more around incremental compiles for
IDE feedback? I didn’t see any issues with the VS Code language server (and I
love the auto format on save, I set this up with prettier and I don’t want to
go back to a place without auto formatters for everything. It’s great DX.)

~~~
tym0
Depends what you're building, I have what I would consider a small server app
(Less than 10 JSON HTTP endpoints with some postgres), a dev incremental build
takes 17.21s and a release build 7m 12s. A similar app in Go would probably
take less than a second to build, they're obviously quite different languages
but that doesn't feel that fast.

~~~
xrisk
Is that because the libraries you link to are gigantic? Even so, the compiler
shouldn’t be compiling those parts of the libraries that aren’t being used,
right?

~~~
pjmlp
So far cargo doesn't do binary dependencies, so when you start a project, or
do CI/CD, it compiles the all world from scratch.

------
drej
This article made rounds recently: speeding up Rust compilation by
replacing/removing dependencies. [https://blog.kodewerx.org/2020/06/the-rust-
compiler-isnt-slo...](https://blog.kodewerx.org/2020/06/the-rust-compiler-
isnt-slow-we-are.html)

~~~
dang
Discussed at
[https://news.ycombinator.com/item?id=23538220](https://news.ycombinator.com/item?id=23538220)

------
uvatbc
Shameless plug: We just started supporting rustc compiler caching and released
it as part of our support for open source projects:
[https://crave.io/#opensource](https://crave.io/#opensource)

We're showcasing the Libra project to start with and would be up for adding
other projects that the community is interested in.

~~~
pdimitar
I'd love to know more, f.ex. you could download and compile the top 100
downloaded Rust crates, demonstrating compiled times with and without Crave.

------
strictfp
Incremental development in Rust isn't particularly slow; it's compiling all
the dependencies which takes the most time, and when you make changes you only
recompile your local crate, which typically is quite fast.

------
kd5bjo
The static linking section here feels a bit weak to me: if a Rust compilation
fails, it usually happens before linking starts. On the other hand, if your
program successfully compiles, do you really care whether the link time
happens at the end of compilation or on program launch?

Unless run-time linking is faster, they both result in the same delay between
the end of code generation and the first execution of your code. Static
linking says you pay that cost once, in the compiler, instead of every program
launch, of which there will be at least one.

~~~
chubot
The work is not equivalent, and that's stated in the article:

 _With dynamic linking the cost of linking is deferred until runtime, and
parts of the linking process can be done lazily, or not at all, as functions
are actually called_

Anecdotally Google's build system Bazel makes good use of the same thing for
tests. The article also mentions that issue:

 _That includes every time you rebuild to run a test._

I don't remember the details since it was long time ago, but dynamic linking
is a huge win here because of the lazy linking. The first function call is
slow but the rest are fast. Loading is faster because you never call most
functions.

Rust seems to have a static dependency problem similar to Google's from 10
years ago (a big pyramid shape rather than using dependency inversion).

\----

edit: related post about the Chrome/Ninja build:

[http://neugierig.org/software/blog/2012/07/gyp-
toc.html](http://neugierig.org/software/blog/2012/07/gyp-toc.html)

 _Here 's one cool hack. You can, by flipping a flag, instead build this tree
of libraries as shared objects. The result isn't something you'd ship to
users, but it speeds up linking time during development considerably as you're
passing smaller subsets of the files to the linker at a time._

 _This leads to a cooler optimization. When you change one source file deep
within the tree, naturally you need to rebuild its object file, then the
library, and then rebuild any binary that depends on the library_ ...

 _So instead, when building a shared object we write out both the shared
object and "table of contents" file (generated with readelf etc.) that lists
all the functions exposed by the resulting library. Then make dependent
binaries only rebuild when the table of contents changes._

------
int_19h
What I don't get is why this:

> rustc is notorious for throwing huge gobs of unoptimized LLVM IR at LLVM and
> expecting LLVM to optimize it all away.

is seen as a problem with Rust, rather than LLVM. Isn't the whole point of
having something like LLVM to have a high-quality backend where all
optimizations only have to be implemented once, and then they just light up
for all the languages that target it?

~~~
johnlorentzson
Some things are best optimized by the actual compiler.

------
crazypython
D skips generating a high-level intermediate representation. (HIR) D has its
own, mature version of Cranelift, the DMD backend, which generates fast
codegen.

Furthermore, like Rust, D has a mature LLVM backend. When compiling D programs
with DMD and LLVM in -O0, LLVM only took slightly longer.

------
Caitin_Chen
This series has 4 episodes. Episodes 1-3:

\- Episode 1: The Rust Compilation Model Calamity
[https://pingcap.com/blog/rust-compilation-model-
calamity](https://pingcap.com/blog/rust-compilation-model-calamity)

\- Episode 2: Generics and Compile-Time in Rust
[https://pingcap.com/blog/generics-and-compile-time-in-
rust](https://pingcap.com/blog/generics-and-compile-time-in-rust)

\- Episode 3: Rust's Huge Compilation Units [https://pingcap.com/blog/rust-
huge-compilation-units](https://pingcap.com/blog/rust-huge-compilation-units)

------
zamalek
> rustc is notorious for throwing huge gobs of unoptimized LLVM IR at LLVM and
> expecting LLVM to optimize it all away.

It might be a good idea to implement a cheap/fast optimization phase according
to rustc assumptions, specifically for rustc, to clean up the mess. Keeping
the rustc emit layer concise and clear should really be a goal, in my opinion.
Obvious code is easier to audit.

------
efnx
But cargo caches well, and after that first compile subsequent compilations
will take mere seconds.

------
floatboth
> In the same experiment I did to calculate the amount of build time spent in
> LLVM, Rust spent 11% of debug build time in the linker

Which linker?

~~~
ndesaulniers
This is a good question; LLD blows the doors off BFD and GOLD IME.

------
zelly
Bazel has rules_rust and supports distributed builds. It might be useful to
switch to that instead of cargo.

------
maxdo
Is it scala slow ?

