
Large Scale Distributed Deep Networks (by Jeff Dean et al) - michael_nielsen
http://research.google.com/archive/large_deep_networks_nips2012.html?
======
michael_nielsen
Also submitted yesterday, by someone else, but it got buried:
<http://news.ycombinator.com/item?id=4775644>

I resubmitted since I'm pretty sure this is of interest to many HN readers.
Examples of why it's interesting include:

1\. Google's deep learning work is now being used to power Android voice
search ([http://googleresearch.blogspot.ca/2012/08/speech-
recognition...](http://googleresearch.blogspot.ca/2012/08/speech-recognition-
and-deep-learning.html) ); and

2\. Dean claims that "We are seeing better than human-level performance in
some visual tasks," in particular, for the problem of extracting house numbers
in photos taken by Google's Street View cars, a job that used to be done by a
large team of people ([http://www.technologyreview.com/news/429442/google-
puts-its-...](http://www.technologyreview.com/news/429442/google-puts-its-
virtual-brain-technology-to-work/) ).

~~~
dumitrue
It's not obvious to me what it means to have "better than human-level
performance" since most of the time the ground-truth itself is defined by
humans :)

~~~
dsl
One example I can think of is a computer could read a house number or a street
sign from 100 feet away, where a human with good vision might be able to make
out the same text at 20 feet.

------
bravura
I did my postdoctoral research on deep learning, and got into it back when
Geoff Hinton's work was just an unpublished tech report.

So if anyone has any questions on it, I will try to answer.

~~~
e98cuenc
Is there any open source ML framework that includes support for DNN?

Do you know of any tutorial that may guide the beginner using DNN, I have no
idea how to choose the number of hidden layers and activation functions.

Thanks!

~~~
seiji
Ask me again in two weeks when my online course in advanced machine learning
for extreme beginners kicks off. We won't start with deep learning
immediately, but it's in the pipeline not too soon after launch.

~~~
e98cuenc
I hope I'm not getting too off-topic, but will this course be taught in
coursera or somewhere else? (I want to put a reminder to check this out).

Thanks!

~~~
seiji
It'll be on a new site for teaching complex things in non-complicated ways.
The goal is to allow everyone from clever middle school students through
retired people to understand the coming changes to the world. There's a huge
on-site community focus too. We don't _want_ there to be 100,000 anonymous
people just going through the motions. There will be plenty of interaction
between course material and community feedback. It's kinda awesome.

Topics will be presented in multiple ways (simple and intermediate) so you can
have plenty of different views on the same material. The material works as
both zero-knowledge intro to the topics as well as quick refreshers if you
haven't seen the material in a while (quick -- what's an eigenvector?!).

The launch courses will be 1.) real-world applications of probability and
statistics (signal extractions), 2.) linear algebra for computer science, and
3.) wildcard (a random assortment of whatever the heck we think is important
or entertaining to know). Future courses are: introduction to neural networks,
introduction to computational neuroscience, introduction to deep learning,
advanced deep learning, how to take over the world with a few dozen GPUs,
avenues by which google will become irrelevant, and robotics for fun and evil.

This is phase zero of a four phase plan. I'll get some pre-launch material
together to shove down HN shortly, then it'll launch a few days later.
Hopefully you'll hear about the project again.

~~~
stephenlee
Great work. It's so cool, can't wait to take part in.

------
samg_
A year ago, I would have no idea what this was about. I'm very thankful that I
live in a world where I can take high quality classes from Coursera that have
given me the foundation to at least understand the abstract. :)

~~~
junto
Which courses did you take out of interest?

I struggle with the mathematics used in neural networks. I can understand code
but as soon as I start to see calculus my brain freezes over. Does anyone know
of a good online course that can give me a crash course in the mathematics
required for neural networks?

My CS bachelors covered this, but I was lazy and drank too much beer. Now 20
years later I want to understand it properly.

~~~
samg_
I've taken Andrew Ng's Machine Learning class, Daphne Koller's Probabilistic
Graphical Models class, Dan Jurafsky and Christopher Manning's Natural
Language Processing class, and currently Geoff Hinton's Neural Networks class.

I have spent a lot of time on Khan Academy to learn the calculus. In my
experience you can get by with a surprisingly small amount of calculus, but it
happens to be a small amount from a high level.

For example, backpropagation is just repeated application of the chain rule.
Did take a while to get a handle on the derivatives, but it's worth it.

------
feniv
This is written by THE Jeff Dean, who is essentially the programming
equivalent of Chuck Norris at Google ( [http://www.quora.com/Jeff-Dean/What-
are-all-the-Jeff-Dean-fa...](http://www.quora.com/Jeff-Dean/What-are-all-the-
Jeff-Dean-facts) )

