
How Chromium Works - aboodman
https://medium.com/@aboodman/in-march-2011-i-drafted-an-article-explaining-how-the-team-responsible-for-google-chrome-ships-c479ba623a1b
======
phasmantistes
Chromium continuous build infrastructure engineer here.

As he says, this is slightly out of date: we forked WebKit to make Blink in
2013, and actually merged Blink into Chromium proper just this week! It is no
longer managed by gclient/DEPS, but instead developed alongside Chromium in
the same repository. But many other things (pdfium, v8, webrtc, and tons of
smaller libraries) are still pulled in via DEPS in exactly the way described.

I'd just like to say this about our development, testing, and release process:
it really really works. We keep moving fast, everyone knows what's going on
throughout the codebase because everyone is working on master, and there are
never any big ugly merge conflicts to resolve when new features land.

There are, of course, problems with the system. We don't have enough resources
to run every test for every commit, so they end up batched and we have to rely
on humans and heuristics to determine which CL in the batch to revert. The set
of configurations on the try-bots doesn't exactly match up with the set of
configurations that are run against tip-of-tree. And flaky tests are, of
course, a perennial problem for everyone. But we're working on making all of
these better, and huge progress has been made in the last couple years.

Ask me anything if you have any questions about our build and test
infrastructure!

~~~
mwcampbell
This is only tangentially related to the topics covered in the article and
your comment, but I'm curious about the Chromium project's decision to mostly
use static linking.

I've always found it interesting that Chrome is shipped mostly as one big
binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and
Safari use several (one for the rendering engine, one for the JS engine, one
or two for networking, etc.). My guess is that Chrome uses one big binary to
minimize startup time, since the OS can just map that binary into memory and
go. Am I correct about the rationale? Another way to ask the same thing is
this: what would be the downsides of using a component build for releases and
not just for development? If you used a component build for releases, you'd
have greater parity between dev and production, and that's generally a good
thing.

I also notice that there are a few modules that get their own DLLs, at least
on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll,
libexif.dll, and libglesv2.dll; I'd think these libraries could be statically
linked just like the few hundred others. Are there good reasons for these
exceptions? Or are these just little details that tend to get overlooked given
the rapid pace of development?

~~~
justinschuh
Like Aaron mentioned, the renderer was originally split into a separate DLL
because we hit the limits of Microsoft's linker at the time. Later, we ended
up relying on that split for disabling win32k inside the sandbox (otherwise we
would have had to deal with a bunch of broken USER and GDI initializations).

As for the remaining DLLs, it mostly comes down to licensing concerns. We
don't include proprietary code in our main binaries, and in order to simplify
IP concerns it's often best to split such code into other DLLs. For example,
you asked about libegl.dll and libglesv2.dll, which I believe include a
licensed copy of the SwiftShader library.

It's also worth pointing out that the performance impact of having a single,
static binary in Windows is much less significant than it used to be. So, were
we starting Chrome today I'm not so sure we'd make the same decision.

------
intruder
I've contributed to Chromium recently and was amazed by how quickly and
painlessly everything was.

I expected to run into a lot of problems with building, missing dependencies
etc. but it just worked. This despite the fact that the fully checked out
source is 37Gb!

I can highly encourage beginners to take a stab at contributing to Chromium,
you can learn a lot.

------
z3t4
This was a positive read! About the stress and modularization. It's important
that developers feel they are an important part of the team. But if they are
working for free, it needs to be fun and not feel like work. Interacting with
others are fun! But not if it gets out of control. So when you modularize, you
should try to make a balance for human interaction in mind. Maybe keep the
number of module interaction down below five at any given time, example: When
a new feature is added, max five modules should need updating.

Sometimes we forget that we are making software for humans, by humans, and not
for compilers and computers.

------
jhallenworld
How does Google get away with not ever supporting previous releases? Don't
customers demand this?

~~~
phasmantistes
A nit to pick here: Google releases Chrome, not Chromium. We only provide
Chromium as a source repository, not binaries. Other independent entities
(such as Linux distros) can provide releases of Chromium, but Google does not.

That said, Google doesn't support previous releases for security reasons. We
are constantly improving the security profile of Chrome, and hackers (both
black hat and white hat) are constantly finding exploits in older versions.
(The same is true for all the other major browsers on the market.)

If you want to use Chrome on a previous/older release, please don't. If you
want to use Chromium on a previous/older release, also please don't, but you
can compile it yourself if you're so inclined.

And for the most part, no, customers don't demand stable old releases of
Chrome. They've learned how bad of a security and usability mistake that is
from IE6, and they don't want to repeat past mistakes any more than we do.

~~~
dragonwriter
> And for the most part, no, customers don't demand stable old releases of
> Chrome. They've learned how bad of a security and usability mistake that is
> from IE6, and they don't want to repeat past mistakes any more than we do.

I suspect that there are lots of enterprise customers that _haven 't_ learned
that ( _encountered_ it, but not _learned_ it), but they probably aren't
_Chrome_ customers, since they are still using IE.

~~~
bengoodger
While we acknowledge some people might think they want this feature, it's been
something we've been unexcited about helping them with it for the reasons
stated. Having the ability to quickly deploy security fixes is fundamental to
Chrome's ethos. This is as close to a point of religion for the Chrome team as
you can get.

~~~
tyingq
I suspect IT customers actually wanted something in between the two extremes.
Your existing IT customers may feel okay about it now, but could change their
minds if, for example, a fairly drastic UI change is pushed out.

------
gghh
feature toggles: do you use them?

my experience with projects that do trunk based development, i.e. a single
branch and time-based releases, is that not all features can be ready when the
release bell "rings", and something is necessarily half-baked. So you end up
enclosing some code into "if (FEATURE_READY) {...}" so that it's there, but it
isn't executed until a few releases later.

This carries some problems, like the explosion of the quantity of toggles over
time, dead code that nobody cleans up, etc.

~~~
phasmantistes
As the article states, Chromium uses run-time checks (feature toggles)
extensively. This lets half-baked features land and start undergoing testing
before the feature is ready to be shown to the world.

But we don't say "oh no, this feature isn't ready, let's wrap it in checks".
The feature is wrapped in checks from the very first commit, so that
everything stays stable on master and on any release branches which are cut
from master.

Then, when the feature is ready for prime-time, the developers working on it
go through and remove the checks. So they don't accumulate in the codebase.

~~~
lmm
Don't you end up with exponentially many combinations to test?

~~~
aboodman
The feature team is responsible for ensuring their in-progress work is
sufficiently tested. This usually means that all their tests are run from the
beginning of the features life. So, we don't end up testing every combination
of features, but instead the union of all features.

With good software design practices we don't usually find bugs that only occur
with some specific non-total combination of features. But of course it does
happen.

The article isn't a way to make provably correct software; it's a way to keep
the wheels on a massive, fast-moving project -- barely :).

~~~
lmm
Sure, which is why I'm interested - I work on a massive, fast-moving
(proprietary) project, and team and feature branches are a vital part of the
workflow, and merges seem less costly than the risks of feature flags.

(Of course, the Knight incident being fresh in our minds, our assessment of
those risks might not be entirely objective)

------
jhallenworld
>We run a public buildbot that constantly runs new changes to our code against
our test suite on every configuration...

In perforce terminology how is this done? You could use branches, but you
could also shelve the change and have the build-bot unshelve it and test.

>We also have a commit queue, which is a way to try a change and have it
landed automatically if the try succeeds

What happens if the change becomes incompatible due to other changes over the
course of running the test? I assume the commit just fails in this case..

~~~
phasmantistes
Well, we don't use perforce (we use Git), so I assume you're asking for your
own benefit.

The buildbot masters watch the source repository for new commits. When they
see a new commit appear, they tell all of their builders (each one
representing a different build configuration) to do a build+test sync'ed to
the new commit hash. The builders check out the repository at that revision,
compile, and upload the binaries for the testers. The testers then download
those binaries, test, and report results. Then we have a display
(build.chromium.org) which shows the results of the compiles and tests on each
configuration for all of the recent commits.

It's actually pretty rare for a patch to become unlandable while the tests are
running. But if it does, it just sends a message to the owner of the change
that says "Hey, your patch failed to land due to merge conflicts. Please
rebase and fix them, then try again."

------
ausjke
Maybe it's time to try chrome again, it felt heavy and memory-hungry for me
and I always ended up with using FF solely.

------
dgemm
Can you click this link for me? [http://a/%%30%30](http://a/%%30%30)

------
gcb0
how chromium works after google jumped in: ignoring users and maximizing
google monetization.

e.g.
[https://code.google.com/p/chromium/issues/detail?id=84177](https://code.google.com/p/chromium/issues/detail?id=84177)

(i also tried to find all the commits or issues where all the 3 ways to
disable sending referrer headers were removed over time, but obviously, there
are no issues and the commits are probably mixed up with other things)

~~~
andrewguenther
\--no-referrers works just fine....

~~~
gcb0
that one have been in and out the code base several times

