I'd love to have cachable builds but I wonder how much effort it takes to maintain such a setup - especially in a small team very far away from silicon valley or google were nobody saw bazel before. Would be a perfect fit for my former company but even there I couldn't convince my colleagues - they build a monster using maven which would be more elegant in bazel but maven integration worked different than the maven algorithm at least 2 years ago...so my PoC turned into reading source code of dependency resolution algorithm in the used scala tool coursier... in the end I had to give up after patching maven issues in several projects we build upon - something nobody is willing to do I guess - to be fair I tried to incorporate Alfresco ECM which is quite an enterprise Java monster on it's own.
Also I couldn't find a good approach to use multiple git repositories almost all tutorials use a monorepo.
Was your project built in multiple languages and tied together using Maven? Or was it just JDK languages/Java?
Tools like Bazel or Nix or <whatever you enjoy> are incredible for many reasons but they are, to a first approximation, very, very high-effort and complex tools that require care. It's like a racing car. You need an engineer on hand to keep the car running, and someone to drive it too. Maybe you're both of those people, but only a driver isn't enough.
They also suffer from another problem which is that most people don't care as much as you or I do. :) So to make it a slam dunk, you have to make a pretty clear case that it's not "just" 2x better, but actually 10x better. And it's actually even harder than that: it can't just be 10x better in practice, it has to appear 10x better at a glance. Like, at the window shopping phase, it needs to catch their eye. That's difficult to nail and basically impossible to do perfectly.
Personally? I'm all-in on "whole sale" build tools like this that hit many nails with one hammer. They solve a tremendous amount of issues. But they aren't always good and I can really see why they can fail to gain traction for many teams: they simply aren't necessary for them, even if they're really a lot better in many ways. I get it.
> Was your project built in multiple languages and tied together using Maven? Or was it just JDK languages/Java?
Exactly that - for one it was a plugin for Alfresco ECM that disobeyed every Alfresco SDK rule :), then a custom Angular frontend that needs to be build according to the plugin - furthermore the business was of that kind that every customer had a specialized build with customization - Bazel would have solved that in a very neat way - including overriding a few source files for a specific customer.
We tried to build then docker images with maven which was just a shitshow considering our problems - however I'm not there anymore but I'd like to have the exact setup that you describe - the big hammer to don't have to deal with all the stupid things - especially as my background is sysadmin and every problem I don't have to solve anymore is good for me :)
I find it interesting that Bazel and Nix don't seem to complement each other as much as I'd think given that they're both 'high effort, high reward tools which deal with hermetic building'.
They certainly both suffer from issues related to upstream packages doing their own thing.
> They also suffer from another problem which is that most people don't care as much as you or I do. :) ...
I use Nix. I haven't use Bazel.
I think Nix is wonderful for enthusiasts. It takes some time to learn. However, Nix is excellent at dealing with packages. And developers often do things which involve packages. -- e.g. Nix is good for declaring a shell with programs that can be made available.
> They certainly both suffer from issues related to upstream packages doing their own thing.
Having used both Bazel and Nix, I agree they suffer here but with a slightly different take: They suffer because they must reinvent the wheel and provide a new API to mimic the upstream package managers. They're in a constant race to keep pace with best practices outside of their community - and patch their APIs accordingly which become "dirty mirrors", only partially reflecting the intent and functionality of upstream tooling.
But that's kind of the point - they apparently need to take over and constrain the system to fully understand the dependency tree.
I wish there was a Bazel/Nix/Guix build tool that had some barrier between the dependency tree and the developer tools that one uses daily. When I'm building the system as a whole, I don't want to think about cargo vs npm. But when I'm writing code for a given component, the native tooling is the correct abstraction. Is it even theoretically possible to resolve those two ideas without introducing another glue language?
Bazel is definitely worth it if you have a large enough repo. But if you have such a large repo you can and should have a team that's dedicated to CI, builds, etc.
If you have a small team and repo it's probably not worth the effort.
The thing is, if you hire 20 Java devs, at least one of them will already be an advanced Maven user. Same for Python and poetry/pipenv, Javascript/npm, etc.
Just speaking to multiple git repositories: could you use submodules for a conglomerated-monorepo?
Otherwise: you can pull in repos as bazel-file-containing additions. I forget the details, but I believe it's something like https://bazel.build/extending/repo . The troubleshooting-while-developing experience is not great though imo, I'd try submodules first.
Thanks - repository_rule looks good. So it's either git submodules to maintain updates or checksums / commits in repository_rule - I probably need to just play around with both approaches and choose the one that annoys me the least :)
I find it hard to buy into one of these "mega build system" ecosystems because there are so many competing tools that descend from Blaze: Bazel, Pants, Buck, Buck2, etc. The migration path to these systems takes substantial effort, so I wouldn't want to bet on the wrong one and have to do it all again a few years later. I'm content to wait patiently for Bazel or Buck2 to clearly win before I invest.
I have the sense that migrating to any hermetic, cacheable build system will be more effort than migrating from one to another: once a project is “safe” to have everything run in sandboxes, the mechanisms for setting up those sandboxes is less important. For example, once one system has accurate dependency information, that can be used as a base for setting up the second system (and presumably this info can be exported for machine-assisted migration).
Of course, the details of each system differ enough that I imagine it’s not trivial.
It seems to be quite active, with regular new releases. I’m lead to believe that https://www.toolchain.com/ is the main sponsor, but there are maintainers from other companies too. I do not know the details, though.
Maybe check out justbuild [0], the main author used to be a bazel dev and wanted to combine it's strengths with the possibility of multi repos and more possibilities to interact with the dependency tree. Quite interesting and very responsive team.
Aaarg, I hate to always be the person that points out "naming issues", but there is already a pretty popular command runner with the name just: https://github.com/casey/just
I'm not going to try to convince you to use Bazel. I think it's great, but ultimately choosing one build system or another is a big decision that involves weighing a lot of different factors, including what your coworkers are amenable to and have experience with. However I don't think Bazel is that difficult to get started with.
The Bazel docs are admittedly a bit challenging, but the easiest way to learn things when you're first getting started is to just look at other projects using Bazel and see how they do things.
Using multiple git repositories is easy, you just use the git_repository() rule which has been in Bazel since pretty much forever. Nowadays you can also use bzlmod (introduced in bazel 5.x) which is a slightly different approach to this problem. There are a bunch of rules in https://github.com/bazelbuild/bazel-central-registry which you can look at to see how to write your own bzlmod registries.
bzlmod looks amazing! Thank you! I don't need to be convinced but I need to sell it to my team and I must be able to solve our problems in it - imho getting it working solves 90% of annoying problems in this whole ci/cd/container mess and that's why I'd like to push for it.
We use Bazel at work for C++, and while defining targets and dependency is easy, I've really struggled with some stuff I would consider basic, like conditionally changing build flags, especially to external dependencies.
Basically, I wanted to conditionally use sanitizers for a specific target and propagate the sanitizers compiler/linker flags to the whole dependency tree.
For code you own, from what I could find, you need to duplicate each target and each dependency (and their own dependencies) for each possible sanitizer and then select the right one. And for external dependencies, you need to use "Aspects", if I understood correctly, to modify the build dependency graph and inject your flags. There's a question on stackoverflow about it, but the answer is very high level (presumably by a Bazel dev), and unfortunately I didn't have time to dig it further and just abandoned.
I also had a lot of issues on an ARM Mac, including the linker issue that is apparently fixed (hooray) and recurrent crashes.
Given we do not use any of best features of Bazel (remote/distributed compilation, multiple languages, and caching (because we basically spawn a new docker container for each build)), I think CMake would have made more sense.
I think you should be using user-defined transitions (https://bazel.build/extending/config#user-defined-transition... ) to use sanitizers. They essentially duplicate the build graph, and you can set build flags on the transition so that all dependencies will be built with those flags.
Awesome but kind of a bummer to see yet another dependency management tool, especially in the C++ ecosystem. Didn’t read that much of the article, but wonder if there’s interop plans with VCPkg or Conan?
No, they are extremely broken. You can only choose:
- Link everything static.
- Link everything dynamic.
- Link user libs as static and system libs as dynamic.
There is no easy way to link a single user lib static/dynamic without resorting to hacks/workarounds like re-importing the shared library or defining weird intermediate targets. It's completely broken.
# you'll need a c compiler installed, xcode is fine on macos
git clone https://github.com/emidln/bazel_static_dynamic_c_demo
cd bazel_static_dynamic_c_demo
bazel run //:foo
ldd bazel-bin/foo # otool -L bazel-bin/foo if you're on MacOS
*You are the consumer.* You are consuming it in foo. Are you building some straw man consumer who might want to delve into my build and rearrange my libraries with no work at all? You can even do that if you want. Remove the linkstatic line from qux. Now it has two outputs: `libqux.a` (default) and a `libqux.so` (implicit). If you want to ship both of them in a release artifact, you can. If you want to mark some binaries `linkstatic = True` and statically link them you can. If you want to dyanmically link some binaries, you can do that too.
I was demonstrating that you can force some libraries to be only static and still partially link some things static and some dyanmic. If you want to get really into the weeds, you could even affect the link line and individually pick dependencies in a custom rule (that is fully compatible with the rest of the bazel tooling). Almost nobody ever needs to do that, but maybe you want to make only every other dependency dynamic to satisfy some weird slide in a talk.
You can use the "linkstatic" feature on the cc_library level. Then that library will be linked statically while other cc_library's will be dynamically linked.
The logic is backwards though. I may have multiple consumers of a library some of which may want static some of which may want dynamic. You need to create/import new targets to do this even though the original target creates both static and dynamic libs by default.
This is just false. Bazel creates both static and dynamic libs by default for every cc_library. The default is to link static, but you can control this on a per-binary basis. You don't need parallel trees of targets or even to write custom rules to access the non-default output groups that contain the shared library. This all just works out of the box.
> You're assuming you have control over all the deps and can set `linkstatic` on them.
In bazel you have complete control over all of your deps. Ultimate power and ultimate responsibility. Even an external BUILD.bazel can be patched when you bring it in (bazel has provisions for patching stuff from http_archive if you supply a patch). You can even ignore an existing BUILD.bazel and write your own if you only care about part of an external thing. If it's a target in your graph, you can certainly modify it. If there is some internal process that prevents you from modifying your own code, I can't help you. Maybe fix that or go somewhere that gives you the authority to do your job.
NX is a very bazel like for frontend focused repositories.
I think what I struggle with the most is the opinions of the rules / plugins. Either I use it as a core task runner (easy, but lots of config the team has to maintain setting up script entries in packages etc) or I have to write custom plugins for anyting not on the "happy path".
I think thats my general frustration with these tools, they aren't terribly well built for SDK type consumption, they don't expose a core set of APIs directly, instead relying on a plugin system that can sometimes be very opaque
I think that this is a problem with branding which unfortunately Bazel is using. I think remote caching and remote exec are not the best features that Bazel has to offer.
First to answer your question: distcc and ccache are great but they offer a limited amount of flexibility in what can be distributed. Bazel's approach is generalized and operates on this basic set of steps:
1. Configure a "target" (define inputs and outputs)
2. Load the deps for the target into a sandbox.
3. Run a command in that sandbox.
4. Copy the outputs out of the sandbox.
With this approach you can execute anything as it's just running a shell program with some args. For example: lets say you are building a game and you have a process where you take raw models from your designers and compress them into an efficient storage format. This can be a `genrule` in your BUILD file and instantly get rerun whenever anyone submits a new change to the models. Now that it's been run in CI all of your devs will not need to rebuild this on their machines. That's the goal. Basically: it is more generic than caching a specific language's data. You can cache everything.
My favorite part of Bazel is the layer of abstraction. Rather than thinking of build steps, what args to gcc are getting passed, etc you are thinking about libraries, binaries, tests, and other artifacts.
Also, this abstraction of inputs and outputs makes it so it's easy to eventually build higher level abstractions. For example, this should be possible at some point:
cc_binary(
name = "foobar",
srcs = [...],
hdrs = ["foobar.h"]
)
# Automatically generates jni binding
# for a C++ library just by reading the
# header file and binding all types.
java_cc_binding(
name = "foobar_java",
deps = [":foobar"],
namespace = "::foobar",
generate_class = "foobar.FoobarBinding",
)
This doesn't yet exist but it could and it would be pretty amazing. Or, another example, imagine AWS publishing a `lambda_binary` rule which you could pass any `*_binary` and run your binary into which automatically generates a packaged and ready-to-go artifact which you could upload to AWS.
I think the framework to allow others to build on this abstraction and make things take less effort overall is the huge value add.
None, bazel's caching implementation is broken because they don't even know or specify what constituents a build hash/key. See this issue from 2018 that's still open [1].
Probably because building targets with tools outside of the workspace is an antipattern, as it violates hermiticity principles. In fact, Bazel generally makes it quite hard to do this, so anyone who ends up in this scenario must have jumped through many hoops to get there.
I agree that the linked issue is legitimate, but I'd argue that this isn't a problem Bazel itself needs to solve--you should fix your build to be fully hermetic.
Not that I'd recommend it, but if you symlink your system library into the bazel build area, as long as your sandboxing setup don't hose you (or you just turn it off), bazel will track system tools/library in the same way as everything else.
I'd still recommend building everything from scratch (and understanding the relationships and graph of your dependencies), but if your build isn't that complicated and you want to role the dice on UB, this isn't that hard.
As an aside, the most galling part of bazel's cache key calculations has to be that it's up to the individual rules to implement this how they see fit. The rules native to bazel written in java vary wildly compared to starlark-written rules. On thing you (or someone in your org) end up becoming pretty comfortable with while using bazel in anger is RTFC.
> Probably because building targets with tools outside of the workspace is an antipattern, as it violates hermiticity principles.
Nonsense. Nothing forces you to use tools outside of your workspace. CMake just requires you to set CMAKE_<LANG>_COMPILER_LAUNCHER[1] to point to a compiler launchers, which can be anywhere where you see fit, including a random path within your workspace.
People try too hard to come up with excuses for design flaws.
Learn to take others seriously without asking for credentials. Whether or not I use C/C++ is irrelevant. Bazel is flexible, and you can use it however you want, correctly or incorrectly. The principles I've mentioned above are language-agnostic, and are recommended best practice regardless of whatever programming language you are building.
> When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system.
> In order to isolate the build, hermetic builds are insensitive to libraries and other software installed on the local or remote host machine. They depend on specific versions of build tools, such as compilers, and dependencies, such as libraries. This makes the build process self-contained as it doesn't rely on services external to the build environment.
My overarching point is, you should fix your build rather than point blame to the tool's authors because you're using it in an unsupported way.
Many people believe that the "traditional" way of building C/C++ applications is an antipattern. Such a belief is, in fact, a core reason to adopt bazel. If you don't believe that, then bazel may not be for you. It is intentionally opinionated in a way that you aren't.
I'm assuming you're referring to the golang model of statically linking everything. That's not really doable when many popular libraries are (L)GPL'd like glibc and libstdc++. It also doesn't work if you want to provide a shared library and need to be compatible with every possible system. That's not my opinion it's just a deficiency of bazel.
Then you build the lowest-supported-version of GCC and glibc, use that as your toolchain in Bazel, and build a dynamic shared library as normal. Using a system-provided toolchain also works, but you have to build on that system using something like Docker, which is certainly an alternative to Bazel but isn't quite meant to serve the same niche.
Non-hermetic is non-reproducible. I can only produce the same build outputs as you if I use the same toolchain, which essentially implies I must run the same OS and patch level, among a host of smaller impurities that can change the output of a build.
Sometimes this is desirable; for example, if you are packaging for a Linux distribution. But that's not the use case Bazel was invented to serve.
I'd love to have cachable builds but I wonder how much effort it takes to maintain such a setup - especially in a small team very far away from silicon valley or google were nobody saw bazel before. Would be a perfect fit for my former company but even there I couldn't convince my colleagues - they build a monster using maven which would be more elegant in bazel but maven integration worked different than the maven algorithm at least 2 years ago...so my PoC turned into reading source code of dependency resolution algorithm in the used scala tool coursier... in the end I had to give up after patching maven issues in several projects we build upon - something nobody is willing to do I guess - to be fair I tried to incorporate Alfresco ECM which is quite an enterprise Java monster on it's own.
Also I couldn't find a good approach to use multiple git repositories almost all tutorials use a monorepo.