
Jeff Dean’s ML System Architecture Blueprint - trcytony
https://medium.com/syncedreview/google-ai-chief-jeff-deans-ml-system-architecture-blueprint-a358e53c68a5
======
mark_l_watson
I work in this field, but nothing state of the art (relatively simple LSTM
models and GAN models). While I found the article informative, it was also a
little depressing to see how far research goes beyond what I am working on. I
spend about 8 hours a week off-work-hours studying and reading papers and I
find it difficult to keep up.

~~~
antpls
I think it makes more sense to focus on the benchmarks. Benchmarks change less
often than the underlying algorithms/models, and they are easier to follow.

Once a model performs consistently well on a given benchmarks over several
years, then it makes sense to get more into the details.

For example in the NLP field, in 2018, there is a focus on multi-tasks models.
Some studies (don't have the refs at hand, sorry) suggest that different
models generalize differently (and some time better) when trained on several
tasks at once.

Anyway, those papers and models are the result of team of researchers working
on the problem full time, with tons of data at their hands. If any sane
individuals were able to keep up with the state of the art, it wouldn't be a
research field, I guess :-)

------
jahjaylee
Although a neat thought, are the number of papers on ArXiv really something
worth comparing to Moore's law? Like at that point, what can't you compare to
Moore's law...

~~~
randcraw
IMHO, the two are incomparable. The exponential growth rate of Moore's Law was
driven largely by a linear rate of shrinkage in 2D, which drove up clock rates
geometrically as microarchitecture component distances shrank two-fold (until
CMOS' heat finally fought back). ML has no similar geometric driven basis that
will continue to drive its rate of growth superlinear.

I suspect this plot is Dean's way of paying homage to Patterson, since he and
Hennessy were famous for similar plots describing CPU performance in their two
architecture textbooks.

~~~
lallysingh
Each graduating PhD takes on several students. They all publish.

~~~
falcor84
I suspect that only a minority of ML PhD graduates stay in academia. So even
though the growth is exponential, it's probably much closer to 1 than to the
student:professor ratio.

------
kingvash
Google really seems to be leading the pack with investments in silicon (e.g.
TPU and recently announced edge tpu[1]). Other traditional silicon companies
(Nvidia, Intel) seem to get it but I have yet to see investments from other
tech companies (Amazon, FB, Netflix).

[1] [https://techcrunch.com/2018/07/25/google-is-making-a-fast-
sp...](https://techcrunch.com/2018/07/25/google-is-making-a-fast-specialized-
tpu-chip-for-edge-devices-and-a-suite-of-services-to-support-it/))

~~~
zeusk
Microsoft Research is pushing along as well, they just don't publicize it as
much.

Azure has already deployed FPGAs (they believe, being able to deploy and make
changes including changes to workload on the fly is more beneficial than the
efficiency compared to using an ASIC) for networking and accelerated ML
(Project Catapult and Project Brainwave).

tbh, I do agree with using FPGAs over ASICs given the speed at which the tech
is moving. Google has already cycled through 3 versions of the TPU.

~~~
hollerith
and yet all 3 versions of the TPU are ASICs, not FPGAs :)

~~~
zeusk
and? that only supports the argument of FPGAs over ASICs (unless Google is
harboring some deep secret on cheap chip tapeout process).

------
typeformer
The blueprint calls for the AI to train on a data set of everything Jeff Dean
does or thinks for the entirety of his life.

~~~
typeformer
No love for bad Jeff Dean jokes here I guess.

~~~
deboflo
“Jeff Dean once shifted a bit so hard it ended up on another computer.”

~~~
deboflo
“Chuck Norris can kill you. Jeff Dean can kill -9 you.“

~~~
godelmachine
Did not understand. Please explain.

~~~
saagarjha
kill -9 sends SIGKILL to a process, which cannot be caught and results in the
immediate termination of the program (except in a couple very specific cases).

~~~
godelmachine
Thanks :)

------
drewmassey
Interesting. There is a kind of obvious conflict between cloud resources and
hardware intensive applications like ML. The zeitgeist is obviously swinging.

------
ratsimihah
Am I the only one thrown off by Jeff Dean wearing a suit?

~~~
thelastidiot
It's called respecting your audience. Everyone probably looked like from a
1970s movie in the room.

~~~
ratsimihah
Hahaha

------
sytelus
First, -1 to authors for publishing paper in IEEE Macro which is behind
paywall. We need to start mass boycott of all IEEE journals considering they
are as bas as Elsevier but have successfully painted themselves as good guys.
Number of papers in ML that I came across and behind paywall are mostly from
IEEE. In any case, we expect better from authors in Google Brain to agree to
publish behind any paywall!

Second, I fail to see any real takeaway or key new insight. Number of papers
grows exponentially in many fields in initial periods.

