
What does it take to make Google work at scale? - mrry
https://docs.google.com/presentation/d/1OvJStE8aohGeI3y5BcYX8bBHwoHYCPu99A3KTTZElr0/
======
peterwwillis
Keep in mind this is really "How does Google (and other big companies) do it",
not "what does it take", because what it actually takes is much simpler than
custom hardware and custom software. This was all pretty accessible 15 years
ago using available software; the trick was putting it together right.

In general, the maxim of "cache everything, everywhere" is the cheapest way to
gain both speed and availability. Of course you need to have results already
to cache them, which is where all that big data processing comes into play,
but there's still way more optimization going on here than is necessary to get
results like this. You could make a tradeoff and give people newly-processed
results more slowly and still give them mostly what they want, and not have to
maintain so much custom software+hardware infrastructure.

Sometimes I feel like Batty from Blade Runner when I think about how well old
crappy software used to work. _" I've seen things you people wouldn't believe.
Attack ships on fire off the shoulder of Orion. I watched mod_perl apps serve
50 thousand dynamic requests per second with nothing but Apache and MySQL. All
those moments will be lost in time... like tears in rain..."_

------
nickpsecurity
Nice overview. I particularly like how they visualized BigTable and MapReduce.
Will make it easier for some to understand. For those liking Spanner, Google
has exceeded that with their awesome F1 RDBMS (built on Spanner/GFS) below:

[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41344.pdf)

Can't wait for someone to clone this. Come to think of it, has anyone cloned
Spanner? I recall some calling GPS a hack but I thought it was engineering
brilliance. Spanner was some good work, too.

~~~
sargun
The people who are working to it are the Cockroach Labs folks:
[http://www.cockroachlabs.com/](http://www.cockroachlabs.com/)

~~~
nickpsecurity
The Wired article says they're not fully replicating Spanner, esp is time
tricks. However, they are replicating a lot of its other capabilities. So,
that's still a great tech in development. Thanks for the link. :)

Still gotta wait a while for a full Spanner replacement with GPS tricks and
all. Or something even better hopefully.

~~~
sargun
You don't _NEED_ the GPS tricks and all, and realistically you can replace
some of the HLC code in Cockroach with a time network time server
implementation that speaks to GPS / Atomic clocks, if you want.

~~~
nickpsecurity
Good to hear :)

~~~
tschottdorf
Note that Spanner always waits out the clock uncertainty on commit, while
Cockroach doesn't need to do that - by design there are fewer situations in
which the offset really matters. We can still benefit from a good bound in
some situations, in particular you get global linearizability between causally
unrelated transactions by waiting out the maximum clock offset (so just what
Spanner does). BTW, I'm not sure when you would ever need the above guarantee,
but it's cool! (disclaimer: actually working on the project).

~~~
nickpsecurity
Again, appreciate the insight.

------
davis_m
"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and
atomic clocks" \- Google

I think it is crazy that something as simple as keeping time can have such a
complicated solution at that sort of scale.

Every review I hear about working at Google makes me want to stay away, just
like the recent conversations about working at Amazon, but crazy stuff like
this always piques my interest.

~~~
ska

       I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.
    

Time keeping is one of the things that is most often underestimated and
screwed up at any scale, in my experience. Many a subtle bug turns out to be a
poor assumption or misunderstanding about how clocks work. Or calendars.

~~~
zo1
Can you elaborate on the need? What possible reason, short of a very contrived
one, is there for having to keep a large number of machines' clocks in-sync?

And for that matter, why would anyone build any process/system/software that
requires a distributed system's machines to all have their clocks in-sync. I
am baffled.

~~~
pjungwir
Many security protocols use credentials that are only valid for a brief time.
Here is a simple example of a problem caused by my server's time being more
than 10 minutes off from the time on S3:

[http://illuminatedcomputing.com/posts/2015/04/paperclip_expi...](http://illuminatedcomputing.com/posts/2015/04/paperclip_expiring_url_and_request_has_expired/)

~~~
abrookewood
Yep, security is a big one. Authentication & encryption can behave very
unpredictably if your times are not in sync. So, you might not be able to
login to a server, or connect via HTTPS etc

------
i336_
Are these slides attached to a video presentation or something similar? I'd
really like to see the talk this accompanies.

------
david_shaw
_> Wow, this file is really popular! Some tools might be unavailable until the
crowd clears._

Hopefully the irony isn't lost on the readers of this document :)

~~~
DannyBee
What irony?

To me, this isn't irony at all. It shows that google can't defeat the
algorithmic complexity of things like consensus and shared state editing, and
so gracefully degrades functionality.

That shows a very good understanding of scaling - realizing any solution you
choose has limits and tradeoffs, and thinking about and handling those limit
cases sanely ahead-of-time, rather than waiting for it to fall over and hoping
for the best.

~~~
munin
I got an error message saying the document wasn't available at all and to come
back after the "crowd clears" which is much more ironic... after I refreshed a
bunch it became available...

------
jordz
It's presentations like this that give me a that spark to go out and build
crazy-shit. Sometimes you HAVE to reinvent the wheel because wheels just don't
work to move objects the size of the planet the wheel is bound to. I'm not a
fan of re-inventing the wheel by any means but it's this kind of stuff that
makes me love going to work every day and basing ideas off presentations like
this!

~~~
i336_
Well, it isn't every day you get to redefine π :D

------
magicalist
previous discussion on work from the same author:
[https://news.ycombinator.com/item?id=9652528](https://news.ycombinator.com/item?id=9652528)

------
vayarajesh
Is it possible to view the video or the presentation given in which the
speaker talks more in detail explaining the slideS?

~~~
ms705
Hi, author here.

The talk for which I put the slide deck together was given at a summer school
and unfortunately not recorded, but if there is sufficient interest, I might
tape a re-run and upload it. (Though unlikely to have time in the next month,
so it might be a while.)

In the meantime, the original papers (listed in the bibliographies at
[http://malteschwarzkopf.de/research/assets/google-
stack.pdf](http://malteschwarzkopf.de/research/assets/google-stack.pdf) and
[http://malteschwarzkopf.de/research/assets/facebook-
stack.pd...](http://malteschwarzkopf.de/research/assets/facebook-stack.pdf))
have _a lot_ more detail than my (very condensed) slides.

~~~
badtoyx
Please DO re-run the talk and share it with us. It would be quite valuable for
a variety of audience. Thanks in advance!

------
ohitsdom
I understood some of these terms.

------
faragon
Wow. That's amazing. Anyone knows how they do a backup/restore of a GFS
distributed file system? (in case they do backup of some sort)

~~~
xyzzyz
Apart from other people mentioning that distributed file systems have
replication as a part of the architecture, Google also uses tape[1][2].

[1] - [http://www.tested.com/tech/1926-why-google-uses-tape-to-
back...](http://www.tested.com/tech/1926-why-google-uses-tape-to-back-up-all-
your-emails/) [2] -
[http://www.theregister.co.uk/2013/12/29/a_year_of_tape_tittl...](http://www.theregister.co.uk/2013/12/29/a_year_of_tape_tittle_tattle/)

------
yzh
This may seem irrelevant, but as a GPU computing researcher, I'm still
disappointed that GPU still hasn't made its way to be the first citizen at
data centers. I know world of HPC looks much different, but I wonder when GPU
can be vastly used in data centers, and will NVIDIA's Pascal architecture make
a huge impact on this?

~~~
ploxiln
I don't think GPUs will be big with "web services" datacenters anytime soon.

As the slides mentioned, the difference between this web-services stuff and
HPC is that HPC has a much higher compute/data ratio. Said another way, this
web-services stuff is all about moving data, collating it a bit, and less
about intensive mathematical processing of it.

Most of the boxes in the diagrams are datastores. Block stores, object stores,
columnar stores, caches. I hate to say it but... big data. Not (as) big
compute.

------
Thaxll
The thing that might be scary working at Google is that you work with tech
than no-one else uses.

~~~
obeattie
I'm not sure that Google thinks of that as a problem. In fact, I'd argue a lot
of the tech that many companies use nowadays exists _because_ Google pioneered
it.

~~~
nostrademons
I think that he's talking about the employee perspective, not the company
perspective. Google's tech stack is mostly proprietary, which means you can't
take your tech skills with you to a new employer.

...And I haven't found it to be a major problem, being an ex-Googler a little
over a year out. Tech skills are easy to pick up on your own. My current
startup is based on Node.js, a native Android client, and AWS; the one before
it was Django & Heroku. Haven't tried looking for jobs yet - I made enough at
Google to not have to worry about that for a while - but I occasionally get
in-bound interest from big-name, fast-growing startups. Most clueful hiring
managers look for experience with _problems_ , not with _solutions_ , and
Google lets you face problems that the rest of the industry won't deal with
for a while.

~~~
delroth
As a Googler I find it to be the most fun and most attractive part of the job.
I'm not too worried about learning specific technologies. Learning the core
fundamentals of how these technologies work is however critical. But since
Google is (still nowadays) pioneering a lot of that, esp. in the distributed
world, that's not really a problem either.

------
whitenoice
Do young (new to google) googlers get to work on these projects?

------
blablablksheep
There is no mention of SPARK like in-memory compute engine(s) Did i miss it in
my read ? thanks in advance

------
novaleaf
Wish there was audio / text that went along with these slides :(

------
halotrope
Kind of ironic: "Wow, this file is really popular! Some tools might be
unavailable until the crowd clears"

~~~
nostrademons
Incidentally, that's part of scalability: knowing when you can drop
functionality and the user won't care. For Google Docs, it's highly unlikely
that a file with 1000s of viewers is actively being _edited_ by 1000s of
users, and more likely the link just got shared on the Internet. So you can
drop edit functionality for everyone but the original author and they won't
care.

Another example - one that I've heard second-hand - is how Facebook gets
consistency on the news feed. Apparently _your own_ writes to Facebook are
sent separately to a write-aside cache, and then the webserver merges them
back in whenever you view a page. As a result, Facebook is always strongly-
consistent when it comes to _your_ comments (you'll never post something and
then fail to see it show up), but it's only eventually-consistent with respect
to _other peoples '_ comments. But then, you won't know about or care about
the latter, because how would you know that you're not seeing something they
posted?

~~~
patio11
Same at LinkedIn. (Source: presentation by their then-CEO at JavaOne, circa
2009.)

