
Evaluating Bazel for building Firefox - dochtman
https://blog.mozilla.org/nfroyd/2019/10/28/evaluating-bazel-for-building-firefox-part-1/
======
munificent
Hermetic declarative build systems like Bazel feel exactly like sound
statically typed languages to me. It's _really_ hard to get all of your
dependencies precisely specified declaratively. The urge to hack in a little
imperative code or jam in a build step that you just "know" needs to happen in
a certain order is always there and it can be very difficult to satisfy the
taskmaster that is the build system. When you have weird cases like a script
that produces a variable set of output files based on the _content_ of some
input file, it's tough to get that working in the build system.

But... if you climb that cliff, there are many rewards at the top. Like the
article notes, you can reuse intermediate build artifacts between developers
(because it's not possible for those artifacts to accidentally depend on
developer-specific state), you can automatically parallelize (because the
build system can tell declaratively which tasks do no depend on each other)
and you can ask it interesting queries about the dependency graph. That sounds
an awful lot like the runtime safety, performance, and static analysis static
types give you.

The challenge is that for small programs, dynamic types are just easier to get
going with. And _every_ build script starts pretty small. So you have a
plethora of simple, hackable build systems like make that work fine at first
but then get more and more painful as the program grows. With software, there
is at least some customer-visible pain that can enable you to justify a
rewrite in a statically-typed language. But build infrastructure is always a
hidden cost so it's really hard to get the resources to migrate to a different
build language.

Maybe the lesson here is to bite the bullet and start with a hermetic build
system at first.

~~~
adgasf
Except the Bazel syntax is really good. Compiling a few C++ files into a
binary is just:

    
    
        cc_binary(
          name = "app",
          srcs = glob([ "*.hpp", "*.cpp" ]),
        )
    

That's more concise than most build systems AND it's cross-platform.

~~~
mzs
I guess but this is also portable and more people know how to extend it:

    
    
        $ cat Makefile 
        SRCS = foo.cpp bar.cpp baz.cpp
        OBJS = $(SRCS:.cpp=.o)
        PROG = app
        EXT  =
        
        .PHONEY = all clean
        
        all : $(PROG)
        
        LDLIBS = $(OBJS)
        
        $(PROG) : $(PROG:$(EXT)=.cc) $(OBJS)
        
        clean :
                rm -f $(PROG) $(OBJS)
        $ make SRCS="`echo *.cpp`" clean all
        rm -f app a.o b.o c.o
        c++ -O2 -pipe -c a.cpp -o a.o
        c++ -O2 -pipe -c b.cpp -o b.o
        c++ -O2 -pipe -c c.cpp -o c.o
        c++ -O2 -pipe  app.cc a.o b.o c.o -o app
        $

~~~
EricBurnett
I love this example, but more because it shows just how obtuse make is even
for simple programs. There's one self-explanatory line in this whole program
(`SRCS = foo.cpp bar.cpp baz.cpp`), and everything else is magic you'd just
have to cargo-cult forward.

I _think_ that it's also broken, if I copy-paste it: your comment renders with
spaces in the `clean :` stanza, but I believe make requires that to be a tab
character?

While certainly simplistic, the bazel example shows one obscure feature (glob)
that's both still fairly obvious, and unnecessary for a direct comparison to
your example. The rest reads clean, and could be replicated/modified by
someone with no clue fairly straightforwardly.

Don't get me wrong, bazel BUILD files will often hide a whole lot of magic.
But the benefit is, for a newcomer the files they have to interact with on a
day-to-day basis are friendly. Take tensorflow, one of the most complicated
bazel repos I know - browse to a random leaf BUILD file and you'll probably
find it quite readable. (I randomly clicked to
[https://github.com/tensorflow/tensorflow/blob/master/tensorf...](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/sparse/BUILD)
, seems pretty clear - especially when I consider it's compiling kernels for
GPUs, which I know nothing about.)

(Disclaimer: Googler, work closely with the bazel team. I like bazel.)

~~~
mzs
The `SRCS = foo.cpp bar.cpp baz.cpp` was my fav part. I like to put every file
used into a Makefile instead of using globs, but my example also showed how
without editing the file someone can build with globs instead ;) Similar
approach if you need the program to be named winApp.exe instead... That's
another 'beauty' of make, you don't have to edit the Makefile.

There's really not much cargo cult in my example. Everything would be in the
most rudimentary doc or tutorial, certainly less reading than if you needed to
use git for the first time.

And yes there is supposed to be a tab there, some copy-paste thing munged
that.

------
steveklabnik
> The typical solution is to become more conservative in how you build things
> such that you can be sure that Y is always built before X…but typically by
> making the dependency implicit by, say, ordering the build commands Just So,
> and not by actually making the dependency explicit to make itself. Maybe
> specifying the explicit dependency is rather difficult, or maybe somebody
> just wants to make things work. After several rounds of these kind of fixes,
> you wind up with Makefiles that are (probably) correct, but probably not as
> fast as it could be, because you’ve likely serialized build steps that could
> have been executed in parallel. And untangling such systems to the point
> that you can properly parallelize things and that you don’t regress
> correctness can be…challenging.

We had a particularly "fun" instance of this sort of Makefile weirdness back
when we used Make for Rust... [https://github.com/rust-
lang/rust/blob/efec34a95a76212b2324d...](https://github.com/rust-
lang/rust/blob/efec34a95a76212b2324d98f3f6d94a1397c2544/mk/prepare.mk#L44-L51)

(Discussed on HN:
[https://news.ycombinator.com/item?id=13655081](https://news.ycombinator.com/item?id=13655081))

------
adgasf
Before it comes up: Bazel ships with its own copy of the JVM so you do not
need to install Java to use it.

~~~
account42
Is this supposed to be a good thing?

~~~
danpalmer
Is it a bad thing?

That might make the installed size large, but if it's isolated to Bazel,
managed by Bazel, updated by Bazel, etc, then why does it matter whether it's
a JVM, a Python binary, a JS VM, or anything else?

~~~
hannob
> Is it a bad thing?

Yes. It makes verifiability much harder.

E.g. Debian put a lot of work into making a more trustworthy build chain by
introducing reproducible builds. There's no Bazel on Debian. And not on many
other distros. Because getting Bazel to be built itself is a complexity
nightmare. "Here, download this big blob where you have no idea how it's been
built" doesn't exactly help.

~~~
zerkten
Could the Bazel folks better define the composition of their blob? That way
they can distribute their blob, but someone could substitute for another
instance of the JVM. I'm assuming ownership of this lies with Bazel and this
is more of a political/motivation problem than a technical one. Or, is that
incorrect?

~~~
hannob
Sure, this is not impossible.

But there's a simple fact: The more complex your build chain is the more
difficult this is. And Java adds a whole bunch of complexity.

Read into the bug report to get an idea what a nightmarish task this is:
[https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=782654](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=782654)

------
rjsw
Firefox will currently build on more platforms than Bazel does. Getting fed up
of the endless list of new build systems.

~~~
qhwudbebd
I agree, and inflicting Bazel's horrible Java dependency on Firefox would be a
tragedy.

~~~
lima
What's wrong with the Java dependency? There's even a single-binary Bazel
executable[1] that brings its own embedded OpenJDK. My company is using Bazel
and the Java dependency has never been an issue.

Even if you write custom rules, you use the Python-like Skylark language and
never interact with Java code.

[1]:
[https://github.com/bazelbuild/bazel/releases](https://github.com/bazelbuild/bazel/releases)
(the single-binary distribution is bazel-1.1.0-linux-x86_64, it's only 40MB).

~~~
snagglegaggle
When using a source based distro or aiming for reproducible builds you'd need
to pull in Java to produce Bazel.

~~~
pjmlp
And the C compiler springs out of thin air into existence?

~~~
snagglegaggle
No, that's generated also, but is highly reusable whereas some people get
along completely fine without Java (unless you make them pull it in for Bazel
for Firefox).

~~~
pjmlp
So a matter of opinion and language religion.

~~~
snagglegaggle
Well, no. Reproducible builds are a security concern and a source base distro
is highly optimizable and configurable. Just because you do not see the
purpose does not mean one is absent.

~~~
lima
But you can still do reproducible builds with Java. GP refers to wanting to
avoid Java for no good reason, not reproducible builds themselves.

------
ndesaulniers
Of all the build systems I've used, Bazel/Blaze has sucked the least.

------
m0zg
Bazel is excellent on Linux and macOS, but it's pretty bad on Windows. I'm not
sure what (if anything) they're going to do about that. Maybe they'll fix it
on Windows though. That'd be pretty cool.

~~~
NickGerleman
Microsoft internally uses something pretty similar to Bazel. I'm not familiar
enough with the two to fully understand motivations for the divergence. It
supposedly initially ran poorly on Mac, just due to differences in what is
cheap on different platforms. I wonder what kind of inherent performance
differences you would find in something "Windows first" vs "Linux/MacOS first"
[https://github.com/microsoft/BuildXL](https://github.com/microsoft/BuildXL)

~~~
EricBurnett
The differences between bazel and BuildXL are less about platform and more
about philosophy. Bazel is about opinionated builds - declare everything, and
you get fast correct builds. Hard to adopt, but super powerful. BuildXL is
more about meeting codebases where they are - take an existing
msbuild|cmake|dscript build with its under-specified dependencies, and figure
out how to parallelize it safely. This delivers quick wins, but comes with
real costs as well (notably: no caching; labour-intensive magic to guess an
initial seed of deps, with a performance cliff when something changes).

(Disclaimer: Googler, work closely with the bazel team. I like bazel; I like
BuildXL for expanding the space of build ideas, but have long-term concerns it
settles in a local maxima that'll be hard to get out of).

------
boris
I wonder if they considered build2[1]? It seems to tick most of their boxes.
In particular, it has first-class Windows support and, in the upcoming
release, even native Clang-targeting-MSVC support (not the clang-cl
wrapper)[2]. While some of the features they are looking for (like faster
build times which I assume means distributed compilation and caching) are not
there yet, we are working on them (and would welcome help)[3].

[1] [https://build2.org](https://build2.org)

[2]
[https://stage.build2.org/?builds=&cf=windows_10-clang*](https://stage.build2.org/?builds=&cf=windows_10-clang*)

[3]
[https://github.com/build2/README/blob/master/roadmap.md](https://github.com/build2/README/blob/master/roadmap.md)

~~~
dochtman
One of the important parts in the post seems to be that it is not just a C++
build system, there are many other tasks executed through the build system. In
that sense, it seems like build2 isn't a great fit.

~~~
boris
On the contrary, build2 is a language-agnostic, general-purpose build system.
There is the bash module[1], for example. And now that we have support for
dynamically buildable/loadable build system modules, we can develop support
for more languages, code generators, etc., without bundling it with the build
system core. I myself am itching to take a stab at Rust. Hopefully will find
time after the release.

[1] [https://build2.org/build2/doc/build2-build-system-
manual.xht...](https://build2.org/build2/doc/build2-build-system-
manual.xhtml#module-bash)

~~~
qznc
The frontpage starts with "build2 is an open source (MIT), cross-platform
build toolchain for developing and packaging C and C++ code."

My impression about the modules you linked to: A secondary feature because
many C/C++ project need some additional scripting.

~~~
boris
See my reply to a sibling comment.

------
NieDzejkob
I'm surprised to see no mention of Tup [1] in the discussion. Verifying the
dependency graph for maximizing parallelism is its major feature.

[1]:
[http://gittup.org/tup/examples.html](http://gittup.org/tup/examples.html)

~~~
swsieber
Does Tup do more than C? Firefox needs to build some Rust things too.

~~~
andrewflnr
It's a very general tool. IIRC from playing with it, it doesn't give a fig
about whether it's running a compiler at all. I think I was using it for a toy
compiler+nasm, actually...

------
xvilka
The problem of Bazel is the JVM requirement. Why not to use something native
and more lightweight?

~~~
pjmlp
You can compile Java to native code.

And in the case you don't want to, it is hardly any different from shipping a
bunch of .so/.dll with the application.

------
jayd16
I'm trying to wrap my head around whether Bazel is worth looking into for the
kind of work loads I typically work in. There doesn't seem like that much
benefit if you have single language code base and especially if you already
have a single build.

I suppose I see the benefit of this setup if you keep many libraries and want
to pin to the latest instead of, an explicit lib version. If you're in a
multi-repo system you're probably not doing this but I can see how Bazel could
actually make this possible.

Can someone shed some light on the CI story? Can Bazel do code deployment as
well? Is that a smart thing to do with Bazel?

~~~
lima
Bazel is worth looking into as soon as you have dependency graphs that your
language's native build system can't deal with efficiently, or you have
multiple languages that have dependencies on each other's artifacts.

A pure Go codebase? Not worth it.

Personally, I use Bazel even for small projects as soon as they involve things
like generated code or gRPC.

Yes, there are multiple rulesets for deployments, like rules_k8s[1] and
rules_docker[2]. Of course, you can easily build your own custom deployment
pipeline.

[1]:
[https://github.com/bazelbuild/rules_k8s](https://github.com/bazelbuild/rules_k8s)

[2]:
[https://github.com/bazelbuild/rules_docker](https://github.com/bazelbuild/rules_docker)

------
txtsd
Is meson not good?

~~~
adgasf
Meson does not have the same feature set as Bazel. Most notably sandboxing and
remote execution.

~~~
qznc
Other build systems (CMake, Meson, etc) should be able to gain the sandboxing
feature. The remote execution feels more fundamental to me and not so easy to
replicate.

------
hartror
TIL Thunderbird is still a thing.

~~~
pja
It’s still one of the best IMAP clients IMO.

~~~
gspr
mbsync/isync changed my email life.

[http://isync.sourceforge.net/mbsync.html](http://isync.sourceforge.net/mbsync.html)

~~~
NilsIRL
I what way?

What do you use it with?

