
The Friendship That Made Google Huge - adrianhon
https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge
======
NelsonMinar
What a lovely profile. I worked at Google early enough that I got to know Jeff
and Sanjay, although never worked directly with them (to my regret). Every
aspect of this article rings true to me, the way they are, the way they work
together. I always admired how incredibly collaborative they were. They never
competed with each other. Or really with anyone else, either, they just
quietly did excellent work and helped others do their work too.

------
shafte
I'm so glad this this article (unlike so many other representations of Jeff
and Sanjay's work) does not overlook Sanjay's contributions.

I don't know what it is; maybe Sanjay is a weird (read: non-English) name,
maybe it's that computing culture's obsession with lone hackers leaves no room
for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for
not falling into that trap.

~~~
Itaxpica
I think a big part of it is something the article touches on: Jeff is an
extrovert, Sanjay is an introvert. Jeff is the much more visible of the two;
the one who people are more likely to have heard speak personally or to have
dealt with. It sounds like this is how they both prefer it, but it definitely
means that Sanjay's contributions get passed over in the common discourse. I
was also glad to see the article avoid that trap.

~~~
ugh123
You mean like Steve and Woz? j/k :)

------
bertmuthalaly
I was so delighted by this piece and then I got to the byline - James Somers
strikes again!

You might know his writing from “You’re Probably Using the Wrong Dictionary”
or his amazing article about New York’s subway system told through the lens of
the countdown clock:
[http://www.theatlantic.com/technology/archive/2015/11/why-
do...](http://www.theatlantic.com/technology/archive/2015/11/why-dont-we-know-
where-all-the-trains-are/415152/)

Highly recommend all his work: [http://jsomers.net/](http://jsomers.net/)

~~~
misiti3780
seems to be a awesome engineer too.

------
stirner
> Most of the fragile insights that laid the foundation of a new vision
> emerged not when the whole group was together, and not when members worked
> alone, but when they collaborated and responded to one another in pairs

The article gives several examples of this dynamic, but one that came to mind
for me was how Monty Python members created sketches for _Flying Circus_ :

> Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman
> worked as one pair isolated from the others, as did Jones and Palin, while
> Idle wrote alone. After a few days, they would join together with Gilliam,
> critique their scripts, and exchange ideas. Their approach to writing was
> democratic. If the majority found an idea humorous, it was included in the
> show.

[https://en.m.wikipedia.org/wiki/Monty_Python](https://en.m.wikipedia.org/wiki/Monty_Python)

------
sytelus
Interesting tidbits:

 _To solve problems at scale, paradoxically, you have to know the smallest
details_ -Alan Eustace

 _Today, Google’s engineers exist in a Great Chain of Being that begins at
Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of
college; Level 3s often have master’s degrees. Getting to Level 4 takes
several years, or a Ph.D. Most progression stops at Level 5. Level 6
engineers—the top ten per cent—are so capable that they could be said to be
the reason a project succeeds; Level 7s are Level 6s with a long track record.
Principal Engineers, the Level 8s, are associated with a major product or
piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of
with reverence. To become a Google Fellow, a Level 10, is to win an honor that
will follow you for life. Google Fellows are usually the world’s leading
experts in their fields. Jeff and Sanjay are Google Senior Fellows—the
company’s first and only Level 11s._

~~~
akhilcacharya
> Level 2s are fresh out of college; Level 3s often have master’s degrees.
> Getting to Level 4 takes several years, or a Ph.D.

I..don't think this is still accurate. New grad SWEs are L3's.

~~~
habosa
Yeah it's mostly accurate but drop level 1. Lowest I've seen for technical
employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the
point where you have "no expectation if advancement" and is the most common
level for mid-career people.

------
enneff
If you want to see an example of Sanjay's elegant code, take a look at his Go
stream package:
[https://github.com/ghemawat/stream](https://github.com/ghemawat/stream)

~~~
stdplaceholder
Sanjay's "experimental" directory in Google's repo -- a place where users can
commit unreviewed code that are just ideas, or park some code that isn't ripe
yet -- is full of gems.

------
fspacef
Good stuff. Every sport has great partnerships (Jordon - Pippen, Brady -
Gronk) and I always wondered if such a thing would even be possible in SE as
I've felt overly self conscious and inefficient while pair programming.

Though what really makes this my favorite long read of the year is how it
follows their story over the course of Google's rise and the subtle
juxtapositions between early Google of "doors across sawhorses" and today with
"Level 0-11". Given all the recent news with engineer walkouts and protests it
makes you realize that Google have probably done the best job of handling the
scaling challenges of being a unicorn and making the necessary cultural
adjustments as they grew in size.

~~~
warent
Don't forget Sam and Frodo

~~~
ineedasername
Yeah, teams are good but every once in a while you need that lone crazy Tom
Bombadil genius to come along and save your butt from disaster.

~~~
kadal
And sometimes you need eagles

~~~
ineedasername
The really bad times are when your entire universe screaches to a halt until
Brandon Sanderson helps it limp across the finish line.

------
tareqak
After reading this article, I'm surprised that Google doesn't encourage pair
programming / developing developer relationships at the level that Jeff and
Sanjay have here. One might even argue that the manner in which interviewing
and hiring is currently done in the tech industry in general might actively
work against building the sorts of relationships where the output is greater
than the sum of the inputs.

Edit: added a missing "the".

~~~
skybrian
In a way, Google does encourage it by requiring all code to be reviewed before
check in. If you're pair programming, you already have a reviewer and can
check in right away. One-person projects can be more frustrating due to the
delay.

It doesn't seem to be enough to encourage widespread pair programming, though.

~~~
jogjayr
Every company I've worked at as a software engineer, even my first where we
emailed patches to each other like cavepeople (and no we were not using git)
has had code reviews before check in. It's not at all the same thing as pair
programming though.

------
puzzle
Holy moly, a mention of Bogdan! He was a legend of his own. Some of the most
consequential changes, like elevating your service to higher QoS levels, which
something like e.g. Gmail would require, were gated on his approval and his
only. Someone made him a SW Easter egg, but the story already on the Internet
gets some basic facts wrong.

~~~
wsxedcrfv
Do tell! There have got to be a few good ones. He interviewed me for his
current company.

~~~
puzzle
The internal Easter egg was basically a random fortune, which cycled through
his greatest "hits" (#1: "No"). You really had to know where to look.

------
timdellinger
Does anyone have a reference for "In 1966, researchers at the System
Development Corporation discovered that the best programmers were more than
ten times as effective as the worst." ?

I'm seeing the year 1968 at [http://thecomputerboys.com/?tag=black-
art#fn-156-1](http://thecomputerboys.com/?tag=black-art#fn-156-1)

Is there a 1966 reference out there somewhere? Or perhaps the actual study was
done in '66 and didn't make it into press until '68? I didn't pull the
original paper.

(The fact checkers at the New Yorker are typically very sharp.)

edit: I pulled the original paper (10.1145/362851.362858). The studies were
done in 1966!

Interesting to see the genesis of the myth of the 10x programmer.

~~~
thwy12321
"Interesting to see the genesis of the myth of the 10x programmer."

How could it be a myth? Ever see a professional sports player compete against
a non professional player? Why wouldnt this dynamic exist in programming?

~~~
superhuzza
Maybe I'm unimaginative, but in sports where performance is objectively
comparable, I can't think of any where professionals 10x amateurs.

Amateur marathons are about 4 hours, professional are about 2 hours. Amateur
deadlifts are about 500lb, professional are about 1000lb. Amateur long jump
might be 6m, professional almost 9m.

~~~
xapata
Cristiano Ronaldo and Messi are extreme outliers in soccer/football, even
compared to other greats. They're probably 10x. Trouble is amateurs don't play
in the big leagues, so we don't have data on how a normal person would peform.
There is an amusing Ronaldo vs Ronald video. I think it's called "Tested to
the Limit."

Occasionally a 1st league team plays a 3rd league. I saw a video of Napoli vs
Nowhere go 26-0. They scored about as fast as the game could be reset to kick-
off. That's more than 10x.

~~~
jhayward
You are confusing game scoring with some concept of "x times ability". In many
games, someone who is only 10% better than opponents can win close to 100% of
the games.

~~~
xapata
What's your productivity metric? Perhaps having only 10% fewer bugs per hour
makes someone 10x faster at writing complete programs.

------
bigtones
This is a wonderfully told story of two very senior engineers who developed a
lifelong friendship and wrote some of the core of the Google search engine I
use countless times per day - I really enjoyed this piece.

------
hn_throwaway_99
Wow, this article is pretty amazing to me. I have a joke (err, kind of a joke)
that I like to say, "It's never a cosmic ray." What I mean by that is that
when you are debugging and getting incredibly frustrated, right when you feel
that you want to give up because you can't find the bug, that you shouldn't
blame "cosmic rays" because 99.999% of the time it's a logic problem.

Here was a case where _it actually was cosmic rays_ (at least, not a logic
problem), and these guys found it. I also agree with the title and intro of
the article - I think it's very possible that if these 2 exact people hadn't
been at Google at the time than Google's path could have been very different.
Makes me also appreciate how much luck is involved in any successful indeavor.
Not luck that they found the bug (that was hard work), but luck that these 2
guys found themselves working at Google right when Google needed them.

~~~
true_tuna
It’s never a cosmic ray but it’s often bad ram. You don’t have to inspect the
binary to diagnose bad ram. The system would be behaving erratically in other
ways due to regular memory corruption. It’s an amazing story but by no means
the only way it could have played out.

------
Jyaif
“In terms of sexiness, compilers are pretty much as boring as it gets,”

That bit surprised me. For me it doesn't get much sexier than compilers.

~~~
ineedasername
Yeah. Maybe that's the outside view, or view from the surface level folks
concerned with the next shiny framework, but I think the best engineers are
always at least a little enamored with deeper nuts and bolts.

~~~
AlexCoventry
I think he meant the business side isn't going to give a damn about a
compiler.

~~~
ineedasername
That makes sense. At least, outside of massive companies like Google, who
would see the value in a compiler that optimized code for their use even a
fraction of a percent better. Then compilers become the "shiny object" for
them.

------
lowercased
“I don’t know why more people don’t do it,” Sanjay said, of programming with a
partner.

=======

Maybe because most employers won't pay for that. Two people? For one level of
output? You'd have to prove yourself a level 11 googler before most places
would give you the luxury of working as a long-term pair.

~~~
jogjayr
Pivotal Labs has had across-the-board pair programming for years[1]. Although
the folks there are undoubtedly talented, I'm fairly certain most aren't at
Jeff and Sanjay's level.

I don't know why companies don't do it, but it's not cost alone. Here are some
of my hypotheses:

1\. It seems like it would be more expensive - which might deter companies
from thinking about it

2\. If you're not already doing it, it's hard to convince all or even a large
fraction of your engineers to do it. It's a completely alien way of working to
most people

3\. It's not obvious to employees who haven't done it that it's clearly
better. And it might not be - I honestly don't know

4\. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5
days/week - so goodbye flextime, work from home, remote working and all the
other arrangements that people structure their lives around

5\. It must be exhausting to sit with, and talk to, another person constantly
your entire working day

1\. [https://pivotal.io/careers](https://pivotal.io/careers)

~~~
groby_b
There've been studies on the "more expensive" part, and IIRC it's about a 15%
premium in terms of development time - there've been a few studies. Here's one
I could turn up quickly:
[https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia....](https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia.PDF)

Simultaneously, you get a bunch of benefits. (Most importantly, there's no bus
factor of 1 for anything)

~~~
lowercased
i'd think another benefit might be 'better code' overall. has there been any
study about the quality of code from these setups? number of bugs or security
issues, for example?

~~~
groby_b
Also covered in that paper

------
continuations
> The world’s most robust computer systems... used special hardware that could
> tolerate single bit-flips. But Google... bought cheaper computers that
> lacked that feature.

> Together, Jeff and Sanjay wrote code to compensate for the offending
> machines.

I'm guessing the article is referring to Google buying servers with non-ECC
RAM.

How do you write code to compensate for non-ECC RAM?

~~~
quicklime
> When a car goes around a turn, more ground must be covered by the outside
> wheels; likewise, the outer edge of a spinning hard disk moves faster than
> the inner one. Google had moved the most frequently accessed data to the
> outside, so that bits could flow faster under the read-head, but had left
> the inner half empty; Jeff and Sanjay used the space to store preprocessed
> data for common search queries

So Google was a company that was too cheap to buy ECC RAM, but willing to let
half of their hard disks go to waste. I'm struggling to understand the logic
behind this; can anyone explain?

~~~
xapata
RAM is expensive. Disk is cheap.

------
strikelaserclaw
these are the kind of articles i like reading, well written, interesting
subject and good content length. Too many times i read bold headlines on HN,
"Why X is bringing america down", then go to read it and just when they get to
the crux of the matter i find the article has ended smh.

~~~
woodhull
The New Yorker does this very consistently every week at an extraordinary
level of quality.

They will even mail you a copy in exchange for a subscription. It's magical.

~~~
forgot-my-pw
Only $6 for 12 weeks for print + digital. Subscribed!

------
jdpedrie
Site's down for me. Archive link:
[http://archive.is/w4oWL](http://archive.is/w4oWL)

------
nyc111
> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row
> representing an indexed word. Sanjay pointed: a digit that should have been
> a 0 was a 1.

I was curious how he can see "a digit that should have been a 0 was a 1" in a
"thick column of 1s and 0s"? Can you give a specific example. I'm not
questioning that he did but I'd like to know a specific example.

~~~
stdplaceholder
For example the high bit in blocks of ASCII text will always be zero. In a
binary dump it would appear as a column of zeros.

~~~
nyc111
Ok. So you don't need to actually "read" and understand the binary code like
you are reading a programming language. You look for out of place numbers.

------
skybrian
I wonder if more couldn't be done to encourage pair programming and
collaboration in general? What are some other great partnerships?

~~~
Daktest
Would highly recommend checking out `The Undoing Project` by Michael Lewis.
It's a biography (of sorts) about Amos Tversky and Daniel Kahneman that has
excellent stories about their fruitful collaboration in behavioral economics.

------
warunsl
Was surprised to hear the audio version of the story has a different title -
"Binary stars". I think the editorial board went with a more click friendly
title.

~~~
panopticon
That was also the title of the print version. The New Yorker tends to re-title
online "reprints" with less vague/terse/artsy titles.

------
jamestimmins
Anyone know where folks like Peter Norvig and Urs Hölzle fit on the L1-L11
scale? I assumed that all of the early employees/now legends were at the top
level, so I was surprised to see the article state otherwise.

~~~
stdplaceholder
Urs doesn't need a level on the individual contributor scale; he's a Senior
Vice President.

~~~
joshuamorton
Urs however does have one, he's listed as a Google fellow (L10) if you go
looking.

As a general comment, to compare, a director is L8 equivalent, and VP L9, (but
on the management ladder) although at VP+ it gets tricky because there are
subdivisions.

Also worth noting that the management ladder goes up more. But the total
number of people who are >L11 is probably <11, so it's not like there's any
data on how exactly that's broken up.

~~~
dweekly
Director is L8, Sr Director is L9, VP is L10, SVP is L11.

~~~
sshumaker
Not exactly; there are 3 levels of VPs and 3 for SVPs (at least in 2015). It’s
just not publicized.

------
partycoder
It is refreshing to see an article celebrating collaboration.

------
sjg007
This article will queue up a slew of startups that pair program.. perhaps a
good thing!

------
the_arun
There was an earlier thread in HN on them here -
[https://news.ycombinator.com/item?id=16744353](https://news.ycombinator.com/item?id=16744353)

------
hi41
It is such a beautiful story of friendship. Even married couple don’t this
level of match in mind. Only a few a luckily like that. Their partnership has
resulted monumental contributions.

------
techs
Great article! Two powerful minds working on analyzing and solving on same
thread of thought. Kudos to the writer, very well presented.

------
gt2
Serious question - is this an ad? For the company, showing they have a
solution to new growth problems which will keep them a growth company? (custom
chip, architected for AI)

And how they solved the problem before with this team?

------
sandGorgon
Does pair programming only work with friends?

~~~
wpietri
Nah. You can do it with anybody. I've been part of long-lived teams that pair.
But there we rotated pairing partners once or twice a day, so that over the
course of the week you'd pair with everybody.

I think that's much more sustainable for most. Pairing with the same person
every day is sort of like sharing a snowbound cabin with them. There are very
few people I get along with so well that they wouldn't grate after a few
weeks.

------
pumaNpl
Awesome article

------
VikingCoder
"Jeff Dean once shifted a bit so hard it ended up on another computer."

~~~
modeless
"Jeff Dean puts his pants on one leg at a time, but if he had more than two
legs, you'd see that his approach is actually O(log n)."

------
glenrivard
Great article. Glad to see Sanjay get the credit he deserves.

