
How Google's build system works - skybrian
http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html
======
thurn
It's a little annoying that every big tech company has solved this problem in
house in their own way. Google, Amazon, Facebook, Apple, and Microsoft all
have internal build systems that do basically the same things.

~~~
robfig
Are there any open source build systems that work similarly? I looked at
basically everything out there, and none provided this functionality. So my
company ended up rolling our own, based on the Google model.

The particular characteristic that I saw missing was package-level build
files. All the open source guys really wanted you have one big build file that
described how to build everything, rather than bunch of small ones that could
be composed.

~~~
ChuckMcM
This is the problem, no there are not any open source systems which can build
systems like Google has. I think that is in part because there are no open
source projects of the complexity and scale of what Google builds and thus
those teams haven't run into the problems that Google did and had to come up
with their own ways to work around them.

When I worked there Google had 20,000 employees. More than half of them were
engineers, and probably at any given moment in time 5,000 unique users might
type 'build' at their desktop after having checked in a change. That means
5,000 instances get build, 5,000 regression test suites run, following at
least 5,000 different 'views' into the repository. (which sat one some
Perforce servers that nearly glowed from all the work they were being asked to
do).

There are few places where such scale is needed, perhaps a couple of dozen,
and they all have competent engineers so they all come up with their own build
system.

If open source were ever to address this particular challenge, it would have
to be in the creation of some really solid interfaces which would allow a
build system to emerge from the tools, rather than trying to explicitly create
a build system from scratch.

~~~
robfig
That is true, although my comment referred to how the build was configured,
rather than the scaling issues you mention.

Even at my current ~12 developer company, we have too big and interconnected
of a codebase to reasonably manage with ant, or buildr, or anything else I
saw. We won't run into "we can't build our code fast enough" for a long time
still :)

There is just a huge gap in build systems, for teams between ant at the small
end and and google3 at the astoundingly huge end

------
jdefarge
Rumours has Blaze, Google's build tool the article describes, is being open
sourced. At least this information was floating around early 2011.

------
Game_Ender
I need to read more before I really understand, but what was the problem with
something like distcc, ccache and CMake? The two downsides I see to the
distcc/CMake approach is slow dependency resolution (looking at lots of files
instead of just packages), and duplicate coping of files by distcc to remote
machines.

~~~
forgotusername
This is quite a high level overview, so possibly underneath they're using
distcc. This at least used to be true for ad-hoc builds on their corporate
network in previous years.

~~~
slug
Don't know if it still holds true, but they used distcc a few years ago:
[http://google-opensource.blogspot.com/2008/08/distccs-
pump-m...](http://google-opensource.blogspot.com/2008/08/distccs-pump-mode-
new-design-for.html)

I also recommend ccache, but it might run into problems with the pump mode,
although it works fine with regular distcc.

