
Rust's Huge Compilation Units - Bella-Xiang
https://pingcap.com/blog/rust-huge-compilation-units/
======
saurik
The very idea of having compilation unit boundaries look like project
boundaries squicks me :/... the lightweight compilation unit structure C/C++
has--with the ability to elide information but the requirement that it be
prototyped--frankly always seemed like the perfect tradeoff, and it generally
means that projects are able to be compiled in massively parallel environments
as all of the dependencies are explicit. This is why distcc is such a trivial
project to just drop on almost C codebase, and is why I have been able to get
our builds doing like 100x parallel compiles on AWS Lambda (and like, before
someone tries to claim that header file bloat somehow prevents this: it
doesn't, and build engineers do massively parallel build farms for C/C++ all
the time, and it is only in some specific corner cases of C++ projects defined
entirely as templates where this comes up, and even then there are trivial
techniques you can use to help break the translation units apart). Every time
a new language comes out and tries to "fix" translation units by adding tons
of cross-file information dependencies without the explicit definition vs.
declaration difference I kind of roll my eyes as I know there's almost no
chance the build is ever going to fast, and these "we need all the code at
once to even begin to analyze it" languages are the worst :(.

------
_nalply
>In my experience though projects tend to start in a single crate, without
great attention to their internal dependency graph, and once compilation time
becomes an issue, they have already created a spaghetti dependency graph that
is difficult to refactor into smaller crates.

This happened to me too. One important problem to overcome is the orphan rule.
This means: you can only implement a trait you own or for a type you own. If
both the type and the trait are not in your crate, you are out of luck.

So my first attempt failed utterly. It was very disappointing. You see, by
splitting the project in different crates I put one type and one trait in
different crates.

After thinking about this problem for a few weeks I accepted that I needed to
write wrappers, so called newtypes. But this hurt a lot. I managed to abstract
away a lot of the boilerplate by writing macros but still am not very happy
about the complexity.

Growing pains.

~~~
saiojd
I agree. Something must absolutely be done about the orphan rules, IMO they
are Rust's biggest wart. They damage composition in a fundamental way, and
gimp the otherwise extremely expressive trait system.

I find it hard to believe that there exists no set of import declarations
which remove all ambiguity, i.e., allowing orphan traits but requiring that
they get imported explicitly.

~~~
_nalply
I think this is a fundamentally difficult problem. Life is sometimes not nice
and you have to choose. I chose to write newtype wrappers and to accept the
complexity.

> I find it hard to believe [...]

I think you should read up stuff about the orphan rule, then you will
understand better.

~~~
saiojd
Yes it does seem to be a very difficult problem. That being said, some hard
problems deserve hard solutions!

------
randombytes6869
Its not just a Rust problem, I've run into the same thing in Java and I'm sure
it exists elsewhere. The only answer I can come up with is to keep your app
cleanly divided into modules. There's some cognitive overhead but its usually
worthwhile.

I recently divided a 100k line Java app into 5 different modules and it
compiles 4 times faster. The new boundaries have encouraged better code as
well. Suddenly we have things like interfaces and IoC. Before, people just
stuffed things wherever.

~~~
lmilcin
I work with both Java and Rust.

Definitely not the same. With Java, you decide what you want to compile with
your build script. You don't need to compile entire application as a single
unit. If you have to recompile a lot of code when you only make small
modification it is likely because did not use many of available ways to just
recompile the classes that you modified.

Also, with Java some optimizations are pushed to runtime. JIT has access to
all code running even if it wasn't available at compilation time.

~~~
pjmlp
Java also does optimizations at AOT, it is all a matter which toolchain one
uses, plenty of choice since 2000.

------
moomin
Just for the record, orphan instances, although possible in Haskell, are
regarded as a big problem and something that should be avoided as much as
possible. The problem being, you can declare an orphan instance yourself,
later discover you need another library, only to discover it implements the
orphan instance as well, preventing compilation. Worst case scenario is that
the two implementations are semantically different and you no longer have any
idea what your code should do.

(Orphan instances in apps are horrible, orphan instances in libraries run the
risk of making your whole library unusable.)

As useful as they sometimes are, I can't help feeling Rust made the right call
here.

------
pcwalton
In Haskell two libraries that define conflicting orphan instances can't be
linked together. Given the culture around code sharing and crates.io, this
would probably be a big problem for Rust. (As I recall I was the first to
propose forbidding orphan instances, although at the time I didn't realize
they were called that.)

~~~
zenhack
Can you expand on that? What about the culture would make this problematic for
Rust specficially?

It does seem like Rust is just enforcing the rough consensus in the Haskell
community re: best practices.

------
trelonid
>In my experience though projects tend to start in a single crate, without
great attention to their internal dependency graph, and once compilation time
becomes an issue, they have already created a spaghetti dependency graph that
is difficult to refactor into smaller crates.

Maybe there should be a lint for not having modules be mutually dependent?

------
saurik
The URL of this article was changed to the following (so the one linked on
Hacker News as of when I posted this comment is now a 404).

[https://pingcap.com/blog/Rust-s-Huge-Compilation-
Units](https://pingcap.com/blog/Rust-s-Huge-Compilation-Units)

~~~
Bella-Xiang
Sorry, the problem is being fixed now

