
Sccache, Mozilla’s distributed compiler cache, now written in Rust - AndrewDucker
https://blog.mozilla.org/ted/2016/11/21/sccache-mozillas-distributed-compiler-cache-now-written-in-rust/
======
zeotroph
Running this _without_ amazon seems to require replacing the hardcoded
"{}://{}.s3{}.amazonaws.com" url with the location where the s3 compatible
service is running on the local network. Now if that was just a default and
could be overwritten with an env var as well... :)

And minio [1] seems to be the easiest pseudo-s3 there is ($ ./minio server
CacheDir/, done?), or are there better alternatives by now?

1: [https://github.com/minio/minio](https://github.com/minio/minio)

~~~
buckhx
Rust's format macro only takes string literals as a template, so you can't
provide a runtime string template. The solution seems to be adding an external
templating lib, which seems like overkill for a single template.

~~~
steveklabnik
[https://crates.io/crates/strfmt](https://crates.io/crates/strfmt) works too.

~~~
chc
Wouldn't that be an external templating lib?

~~~
steveklabnik
In some sense, yes. In another, it's not some sort of different kind of
templating, it's the same stuff, just calculated at runtime instead of compile
time. I assume the lack of wanting a library for this is due to its weight and
learning some other form of templating; this crate is pretty small, and isn't
different in that way.

------
actuallyalys
Congrats to Mozilla for not only creating a programming language research
project that engaged the community, but also growing it into a successful
language that's useful and robust for real-world projects.

Mozilla sometimes gets flak for its experiments (or abandoning them),
sometimes deservedly, but by doing so many of them and not being afraid to
cancel them, they occasionally get big wins like Rust.

------
jtille
I've been using distcc for quicker distributed builds without issue for many
years. What extra features does sccache bring to the table other than it being
rewritten in Rust?

~~~
tedmielczarek
distcc is great, and as I mentioned in that Reddit comment some of my
colleagues are using icecream to great success.

I won't repeat my entire comment from Reddit, but one other notable point is
that tools like ccache/distcc don't generally support MSVC, and we build
Firefox for Windows with MSVC, so that's pretty important to us. And frankly,
our Windows builds are slow enough that we can use all the build time wins we
can get.

~~~
radarsat1
Color me surprised. As a developer, the number of times "Windows support" or
"MSVC" is the special case that requires you to go through contortions..
_shrug_

It's been very frustrating on multiple open source projects I've worked on,
people proposing to change entire build systems of projects just so that MSVC
can be included as a build target. I hope the new Bash-on-Windows stuff will
eventually make supporting MSVC a little simpler from various Unix-oriented
build tools.

~~~
quotemstr
Meanwhile, GCC works fine on Windows. (Mingw builds normal non Cygwin
programs.) There's no need to bow to cl.exe just because you want to build for
Windows

~~~
tedmielczarek
If you're shipping binaries to end users Microsoft's optimizer is pretty
fantastic, especially if you build with PGO. Plus MinGW sometimes lags behind
with things added in newer Microsoft SDKs, so there can be some friction
there. I think things are getting better with clang-cl, so that might be a
feasible direction in the near future.

------
tedmielczarek
Just as a neat data point, we hadn't been using sccache for Firefox builds in
our new CI infrastructure (Taskcluster) for various reasons. After rolling out
the sccache rewrite I fixed that, and build times in that environment dropped
by 32-38%:
[https://treeherder.mozilla.org/perf.html#/alerts?id=4335](https://treeherder.mozilla.org/perf.html#/alerts?id=4335)

------
jononor
No tests? :(

EDIT: my bad, they were inside src/

~~~
cesarb
In Rust, it's very common to put tests right next to the code; all functions
tagged with #[test] will be run as tests, and examples within the function
documentation comments will also be run as tests by default.

For instance, randomly looking at the first file in that repository,
cache/cache.rs, you can see at the end a submodule tagged with #[cfg(test)]
(meaning it will be compiled only when running the tests), and within it
several functions tagged with #[test].

~~~
christophilus
I can't figure out whether or not I like this convention. I used to work with
a guy who did the same thing with C++ and C# tests: right there alongside the
code they tested. On the one hand, it is nice to have all of the relevant code
in one place for context. On the other hand, it muddied up the actual business
logic with tests, so you had to perform some mental work to figure out whether
the code you were looking at was for production use or only for testing.

~~~
Yoric
Note that Rust lets you put tests in tests/ instead of src/. So typically, you
put short unit tests in src/ and longer/integration tests in tests/. Oh, and
examples that show up in the documentation are also run alongside tests.

Also, the compiler will tell you if you have code that's not labelled with
#[cfg(test)] but is designed to run only in tests.

~~~
tyoverby
The big difference with `tests/` is that they _have_ to be integration tests.
This means that you can't test private functions from there.

~~~
Manishearth
Not .. exactly. An integration test usually tests an entire application at
once. `tests/` can still test things in the public API (indeed, we do this in
Servo a lot) without being an integration test.

The point is that some unit tests cannot be written in tests/, whereas all
integration tests can be.

------
luckydude
I didn't understand his comment on Rust's match expression. Other than it
returning a value (and insisting a default clause) it's just like C's switch,
right?

~~~
Tarean
It is much more like haskell's pattern matching. Notably it handles tagged
unions (in rust enum). Here is a good example from the
[book]([https://doc.rust-lang.org/book/match.html](https://doc.rust-
lang.org/book/match.html)):

    
    
        enum Message {
            Quit,
            ChangeColor(i32, i32, i32),
            Move { x: i32, y: i32 },
            Write(String),
        }
    
        fn quit() { /* ... */ }
        fn change_color(r: i32, g: i32, b: i32) { /* ... */ }
        fn move_cursor(x: i32, y: i32) { /* ... */ }
    
        fn process_message(msg: Message) {
            match msg {
                Message::Quit => quit(),
                Message::ChangeColor(r, g, b) => change_color(r, g, b),
                Message::Move { x: x, y: y } => move_cursor(x, y),
                Message::Write(s) => println!("{}", s),
            };
        }

------
throwaway40483
When I see these rewrites, I'm always left wondering about the speedup
achieved. How much was due to:

1) Going from programming language X to Y 2) Rewriting the algorithm in
language Y

I suspect (without any proof) that too much is attributed to 1) and not enough
to 2).

~~~
sangnoir
3) Removing "unnecessary cruft" during the rewrite, measuring the speedup and
then gradually adding the "cruft" (features) back one by one as previously
unknown boundary conditions are encountered.

Restart the process again after a few years of years, as is tradition.

~~~
tedmielczarek
This is not what happened in this case (although I have seen this pattern in
other situations in the past). The Rust version is a faithful port of the
feature set of the Python version. (It had to be, we were using essentially
the entire feature set in production.)

------
Ericson2314
Parts of Mozilla also use Nix so...

~~~
gmfawcett
...so what, in particular? Don't leave us hanging!

~~~
Ericson2314
It would be nice to use Nix in a more fine-grained way (e.g. per make rule
rather than per package).

------
bpicolo
That's some tiny font.

~~~
nilved
It's using one of Phu Ly's WordPress themes from 10 years ago so that may be
why. I'd recognize his themes anywhere. Back then 8pt was standard and
flexible units were just catching on. I feel old.

