
The Google Stack [pdf] - mrry
http://malteschwarzkopf.de/research/assets/google-stack.pdf
======
mrry
It's interesting to compare this with the Facebook stack, drawn by the same
author:

[http://malteschwarzkopf.de/research/assets/facebook-
stack.pd...](http://malteschwarzkopf.de/research/assets/facebook-stack.pdf)

…and it would be doubly interesting to see the "related work" arrows between
the different companies' infrastructure platforms. We'd need a 3D
visualization for that though.

~~~
aristus
I'd take this chart with a grain of salt, given the FB paper. Peregrine is no
longer in service, IIRC. It was a friendly rival to Scuba, and was eventually
replaced by Presto, which is not even mentioned. Also not mentioned are
several important things like the pub/sub ecosystem built on Scribe. Haystack
is long dead, except for maybe some archived stuff. Lastly, PHP-to-C++ has not
been a thing since early 2012.

~~~
ms705
Original author here.

This is interesting -- I tried to do my best to find out what's still in use
and what is deprecated based on public information; happy to amend if anything
is incorrect. (If you have publicly accessible written sources saying so,
that'd be ideal!)

Note that owing to its origin (as part of my PhD thesis), this chart only
mentions systems about which scientific, peer-reviewed papers have been
published. That's why Scribe and Presto are missing; I couldn't find any
papers about them. For Scribe, the Github repo I found explicitly says that it
is no longer being maintained, although maybe it's still used internally.

Re Haystack: I'm surprised it's deprecated -- the f4 paper (2014) heavily
implies that it is still used for "hot" blobs.

Re HipHop: ok, I wasn't sure if it was still current, since I had heard
somewhere that it's been superseded. Couldn't find anything definite saying
so, though. If you have a pointer, that'd be great.

BTW, one reason I posted these on Twitter was the hope to get exactly this
kind of fact-checking going, so I'm pleased that feedback is coming :-)

~~~
aristus
HipHop's replacement was pretty widely reported:
[http://www.wired.com/2013/06/facebook-hhvm-
saga/](http://www.wired.com/2013/06/facebook-hhvm-saga/)

This link has papers on pub/sub, HHVM, and so on:
[https://research.facebook.com/publications/](https://research.facebook.com/publications/)

Re Haystack: possible I am misremembering, or the project to completely
replace Haystack stalled since I left.

If you want to gather a more complete picture of infrastructure at these
companies I suggest, well, not imposing the strange limitation of only reading
peer-reviewed papers. Almost none of the stuff I worked on ended up in
conference proceedings.

~~~
ms705
Thanks, I've added HHVM and marked HipHop as superseded.

I also added Wormhole, which I think is the pub/sub system you're referring to
(published in NSDI 2015:
[https://www.usenix.org/system/files/conference/nsdi15/nsdi15...](https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-
sharma.pdf)).

Updated version at original URL:
[http://malteschwarzkopf.de/research/assets/facebook-
stack.pd...](http://malteschwarzkopf.de/research/assets/facebook-stack.pdf)

Regarding the focus on academic papers: I agree that this does not reflect the
entirety of what goes on inside companies (or indeed how open they are; FB and
Google also release lots of open-source software). Certainly, only reading the
papers would be a bad idea. However, a peer-reviewed paper (unlike, say, a
blog post) is a permanent, citable reference and is part of the scientific
literature. This sets a quality bar (enforced through peer review, which
deemed the description to be plausible and accurate), and allows the amount of
information to remain manageable. The number of other sources of information
makes them impractical to write up concisely, and it is hard to say what ought
to be included and what should not when going beyond published papers.

I don't think anyone should base their perception of how Google's or
Facebook's stack works on these charts and bibliographies -- not least because
they will quickly be out of date. However, I personally find them useful as a
quick reference to comprehensive, high-quality descriptions of systems that
are regularly mentioned in discussions :-)

~~~
Maro
Other ways to get information:

1\. Ask the developers per email.

2\. Fly out to SF, visit the campus, have lunch.

Will work for FB (I do it all the time), Google people won't tell you
anything.

------
wwweston
When I'm looking at this diagram I'm struck by how much it diverges from the
typical entities invoked when we generally describe a tech stack.

There's almost no discussion of language or framework or "patterns" in the
software idiom sense. Instead it's largely a wide variety of data stores and
access layers.

Bad programmers worry about code, good programmers worry about data
structures?

~~~
endtime
I think it's a matter of scale, rather than quality - we obviously use
languages and frameworks and patterns at Google.

~~~
wwweston
Scale of operations at Google, or conceptual scale of the diagram?

And if you feel like language and framework choice plays a significant enough
role at Google that "stack" in the more traditional sense is something that's
carefully considered by engineering, I'd love to hear about the details.

~~~
endtime
The scale of the diagram, though obviously a diagram of that scale can only be
drawn for an organization with operations at that scale - so kind of both.

I've mostly done backend/data stuff at Google, so I can't speak to traditional
web dev decisions, but I've written design docs which discuss the tradeoffs
between using Bigtable and Spanner, or Flume vs. Mapreduce, or one serving
strategy vs. another, for some specific thing. Maybe those choices are vaguely
analogous to choosing between Postgres or Mongo, nginx or Apache, etc. I
imagine the guys who write webapps (not Search, obviously, but internal apps
or things like our help pages) consider whether to use App Engine, Angular,
Django, etc. on a per-app basis.

------
vgt
Interesting is that much of this is externalized through Google Cloud in one
way or another:

\- GFS/Colossus = GCS

\- Borg = Kubernetes

\- Dremel = BigQuery

\- BigTable = BigTable (naturally)

\- FlumeJava,MapReduce,MillWheel = Dataflow

~~~
exacube
GFS/Colossues are distributed file systems, which GCS doesn't really address
(I think GCS is akin to AWS' S3?)

~~~
ecnahc515
Yep, but GCS is probably powered by GFS

~~~
thrownaway2424
GFS is so dead few people even remember using it. But to say that something is
"based on GFS" or "based on Colossus" is to say very little. If it stores data
it is eventually "based on Colossus". You could say just as much if you said
it was "based on ext4".

------
amelius
Google has many services. This diagram means little if it does not specify
what services use what parts of the stack.

~~~
johannes1234321
All services need to process and store large amounts of data. For that they
need the building blocks firming the core stack which most projects (YouTube
seems to be the big exception from what I hear) share. That is shown.

------
thrownaway2424
An interesting exercise in blind men describing the elephant.

~~~
mrry
Malte Schwarzkopf is the first author of the Omega paper:

[http://eurosys2013.tudos.org/wp-
content/uploads/2013/paper/S...](http://eurosys2013.tudos.org/wp-
content/uploads/2013/paper/Schwarzkopf.pdf)

...so not entirely blind.

~~~
thrownaway2424
One of the reasons Omega failed was because its authors went up in an ivory
tower and ignored everything about what was actually happening in Google
datacenters for years. That a diagram this full of WTF could be produced by
one of the Omega authors does not surprise me.

------
yukinon
It looks like there might be some missing parts to this. I was under the
impression Google has a layer of MariaDB databases somewhere.

~~~
dpe82
Are you thinking of this?
[https://github.com/youtube/vitess](https://github.com/youtube/vitess)

~~~
yukinon
That is definitely it, it looks like Youtube uses MySQL/MariaDB. Thanks for
the link.

------
Xorlev
How about Mesa?

