
Jeff Dean’s Lecture for YC AI [video] - danicgross
http://blog.ycombinator.com/jeff-deans-lecture-for-yc-ai?src=hn
======
iandanforth
The notion of running one giant model that has many sub-talents is epic. I can
imagine that all the disparate models they run today could fuse into a giant
network that melds predictions and guides computation as required by the task.
That seems like a very Jeff Dean scale endeavor.

~~~
cosminro
He is referring to this paper
[https://research.googleblog.com/2017/06/multimodel-multi-
tas...](https://research.googleblog.com/2017/06/multimodel-multi-task-machine-
learning.html)

Where they used one model to do image recognition, speech recognition and
translation in the same network.

~~~
shpx
He also mentions sparsely activating only the neurons that matter, which they
explore in
[https://arxiv.org/pdf/1701.06538.pdf](https://arxiv.org/pdf/1701.06538.pdf)

Personally I didn't find it very satisfying, I imagine something more
fundamental and self referential.

------
litzer
As somebody who's recently starting to learn more about ML, a lot of the work
of an ML engineer does seem to be automate-able (not doing research or pushing
boundaries but just applying ML to some product need). For example, choosing
hyperparameters, evaluating which features to collect, etc seem to be things
that can be automated with very little human input.

His slide on "learning to learn" has a goal of removing the ML expert in the
equation. Can somebody who's more of an expert in the field comment on how
plausible it is? Specifically, in the near future, will we only need ML people
who do research, due to the application being so trivial to do once automated?

~~~
halflings
You will still need data engineers to build the whole data ingestion and
processing pipeline (although that can be easy if standardised tools are
available, such as spark, it's still a challenge in many cases).

~~~
litzer
Right, but I'd consider that falling closer to the realm of general software
engineering -- similar to tasks of collecting analytics of users or building
infrastructure to get data from point A to point B.

Maybe that currently is some parts of the job of an ML engineer. But if that's
the _only_ part, I don't think that role should be called one of ML engineer
anymore

------
sputknick
If Tensorflow becomes the default library for Deep learning, is this a good
thing or bad thing? Does it help in that all researchers can focus on what's
important (the data and results) or does it hurt in that Google now controls
an important paradigm for the next generation of computing?

~~~
blueyes
PyTorch is becoming much more popular among researchers, even if TensorFlow
use is widespread among data scientists.

Would it be bad if TensorFlow became the default library for deep learning?
For companies that aren't Google, yes. Many organizations don't like how
Google's controlled Android development.

But I think an important part of Dean's lecture are the slides that talk about
making ML/DL expertise obsolete. That's the future, whether you use TensorFlow
or not.

~~~
deepnotderp
It's cute, but isn't going to completely obsolete ML/DL expertise. (And
believe me, as a prospective supplier of DL compute, I would love for the
answer to be "more compute").

To give you an idea of how much compute it took, they spent _two million
dollars_ for one run on a relatively toy dataset, CIFAR-10. Imagine how much
it would take on an imagenet sized dataset! Can your company afford ~$20
million+ per dataset?

I _do_ think that hyperparameter twiddling aspect might get automated, but
believe me, that is much welcomed by the DL research community! I would much
rather spend my time on new ideas rather than trying ten different
initializations :)

~~~
blueyes
Sure, but that's $2m of today's compute on today's chips. The constraints on
chips and cost are moving, and Google is pushing them.

~~~
deepnotderp
Sure, so it becomes ~$2 million for an imagenet scale dataset with TPUs and
~$200K with our chips. Still pretty expensive :)

------
hallman76
As a ML enthusiast, this is incredible to watch!

I'm completely blown away that Google was working on full-scale physical
architectures that were optimized for these problems. Talk about being two
steps ahead of the game!

~~~
londons_explore
Smart people, combined with piles of cash, gets you ahead at almost
anything...

Sadly there's still a lot of smart people without access to cash (academics),
and piles of cash without smart people to get the value out of it.

------
bluetwo
If a doctor misdiagnosis an eye ailment, they might end up with a malpractice
lawsuit. If an ML program misdiagnoses an eye ailment, what is going to
happen?

~~~
otoburb
The commercial entity backing the ML program should be responsible. If one
doesn't exist then the comparison fails, as one would presumably not be
willing to trust the competency of a public domain ML program without a true,
commercially and medically backed "second" opinion (algorithmic or otherwise).

Perhaps one day, ML programs running eye diagnostics will be as cheap and
disposable to use as pregnancy stick tests: "Point and diagnose."

~~~
TuringNYC
Yes, the commercial entity backing the ML program, and possibly multiple
associated entities would be held responsible, or at least dragged into
litigation. You can get insurance companies to write custom policies for such
things, but I dont think there has yet been a standards-setting case -- anyone
know of existing or in-progress case law in this arena?

------
deboflo
Once, in early 2002, when the index servers went down, Jeff Dean answered user
queries manually for two hours. Evals showed a quality improvement of 5
points.

~~~
1_2__4
I never thought Jeff Dean jokes would be a shibboleth but here we are.

~~~
nickelbox
I could imagine there being external Jeff Dean fans, but yeah, that jargon

~~~
ma2rten
[https://www.quora.com/What-are-all-the-Jeff-Dean-
facts](https://www.quora.com/What-are-all-the-Jeff-Dean-facts)

