
Programmers’ Build Errors: A Case Study at Google [pdf] - Fr3dd1
http://static.googleusercontent.com/media/research.google.com/de//pubs/archive/42184.pdf
======
zaphar
As a recent employee of Google, Google's build system is the one tool I miss
the most. No other available tool outside of google even comes close.

~~~
gecko
Can you explain why? I pretty much hate all the ones I've worked with; some
insight into a system that people actually enjoy to use would be really
valuable.

~~~
MrBuddyCasino
Yes please. After spending way to much time with Ant / Maven / Gradle, I'm
convinced they all got it wrong. A build system that doesn't suck is like a
unicorn - rumored to exist, but nobody's seen one yet (except Leiningen,
people seem to have no complaints about that one).

~~~
stonemetal
From what I understand Leiningen is Maven in lisp clothing. Any complaints you
have with Maven might apply to Lein as well, unless it is just the XML you
don't like. I have only used it a little, but so far it seems nice.

------
sanxiyn
Findings I found interesting:

1\. No correlation between build frequency and build failure ratio. 2. No
correlation between build time and build frequency. 3. No correlation between
experience and build failure ratio.

~~~
Fr3dd1
For 3. you have to look at the difinition of an experienced and a novice
developer they used in this study. I think maybe for google the definition
fits but not in its general use.

~~~
Roboprog
Agreed. "Experienced" seemed a dubious definition. Experience != Skill, as
well.

One of the things in the paper was that there was a population with middling
errors that did "many" builds, and a bifurcation of 2 populations that did
"few" builds - one with few errors, and one with many errors. (pg 9, section
4.1 "How often do builds fail?")

Could it be that there is a group of gurus who don't need to constantly build,
and which batch up testing of larger feature sets, and another group of
perpetual freshmen who spend so much time fixing their fault riddled crap that
they simply cannot submit very many builds? Yes, I know they attributed at
least some of this to the employee role being done, but it makes me wonder.
"Experience" would not tease this pattern out. (while it seems to me customary
to build more often in the initial stages of using a new language, we've all
worked with "that guy" who just bumbles through, year after year)

------
cpeterso
I think Mozilla's Rust team was considering telemetry reports, not just for
rustc compiler crashes, but also for syntax errors in user code. Knowing what
kind of syntax errors are most common in user code could provide feedback for
language design.

~~~
numbsafari
That would be really cool, especially if lots of OSS projects where to enable
such reporting.

Rather than building it into the rust tooling, it'd be cool if they built a
collection service and then provided hooks into things like Jenkins / Travis
CI that are being used by OSS projects. That way you could gather data
regardless of the language used.

------
lifeisstillgood
I have been mulling over a research MSc looking at the metrics available from
OSS projects and how they might feed into better software practises (I think
we can improve somewhere).

This stuff kind of inspires me - but also makes me quake a little in my boots

------
seanbanana
Link failure, anyone have a copy?

~~~
Fr3dd1
Just tested it, the links work fine.

------
dk8996
So Java > C++ ?

~~~
Roboprog
C++ is a pretty easy target to pick on, though. I'm convinced that it is pure
machismo that makes anybody use it, rather than a desire for productivity.

Of note on pg 9 is the "resolution time" graph. Look at the time to fix
"missing right parentheses". While the median time to fix this C++ error is
about 6 or 7 minutes, about 25% of the time it takes so long that the (75
percentile) edge of the box is off the chart - over 50 minutes. (disclaimer -
it doesn't say how _often_ this error happens)

What kind of masochist writes such a giant run-on expression that it takes an
hour to figure out where to match up the parentheses??? (and doesn't know how
to use the parentheses matcher in the editor)

Perhaps there is an untold story for that error, or perhaps C++ just attracts
a lot of people who like a challenge.

...

Turbo Pascal (5.5 and above) > C++ as well, but Java was free, and ran on
unix.

