

Andrew Ng and the Quest for the New AI - ivoflipse
http://www.wired.com/wiredenterprise/2013/05/neuro-artificial-intelligence/

======
jhartmann
I've said this before, but deep learning is terribly powerful precisely
because you don't have to spend lots of time doing feature engineering. Multi
layer networks that are trained in semi-supervised, unsupervised and
supervised fashions all can now produce networks that meet or beat the state
of the art hand created models for speech, handwriting recognition, ocr, and
object recognition. We are only just beginning to see what is possible with
these sorts of techniques. I predict within a few years time we will see a
huge renaissance in AI research and Neural Network research specifically as
these techniques are applied more broadly in industry. My startup is building
some cool stuff around this technology, and I know there are hundreds like me
out there. This is going to be a fun ride.

~~~
wookietrader
This has been said about neural nets two times already. Sadly, they did never
deliver.

There are still applications where e.g. random forests beat the crap out of
all kinds of deep learning algorithms in (a) training time (b) predictive
quality (c) prediction time.

We should stop hyping this. I am a researcher working in deep learning myself,
but the current deep learning hype is actually what makes me worry that I will
have trouble getting a job because industry will be disappointed a third time.

~~~
jhartmann
Well I think a big part of this is that right now we have finally gotten to
where the algorithms + the required computing power are starting to become
more widely available. Cheap graphics cards or things like Intel's Phi +
techniques like drop out to prevent overfitting are really enabling much more
sophisticated things to be done in a reasonable wall time. Granted multilayer
neural networks aren't a free lunch that will solve everything, but there are
large classes of problems that are falling to these techniques all the time.
We are also finding that Neural Networks scale very well to very large
architectures, better than some of the other techniques. I understand we
should be careful not to overhype since we have seen previous excitement cause
a mass exodus from this research before. I however think people like Hinton
were always right, and this was awesome stuff. We just couldn't really take
advantage of it because we could never train it for long enough and we hadn't
learned how to do things efficiently yet.

~~~
wookietrader
Yes you are right.

Still, deep learning has done nothing more than classification right now.

What about predictive distributions, regression of complicated outputs (e.g.
periodic data) and, most of all, heterogenous inputs? Right: nothing
impressive has been done in that area, despite of huge amounts of practical
problems.

Let's see if deep learning generalizes to those things. If it does (and I
personally believe so) let's be happy. Before that, we still have to envy what
Gaussian processes, Gradient boosting machines and random forests can do what
DL so far cannot.

~~~
imurray
_Still, deep learning has done nothing more than classification right now.
What about predictive distributions, regression of complicated outputs..._

<http://homepages.inf.ed.ac.uk/imurray2/pub/12deepai/> has predictive
distributions from deep learning, passed on to time-series smoothing for
articulatory inversion. It's a previous neural net approach made deep, and
working better as a result.

(I agree that like any machine learning framework, neural networks have their
strengths and weaknesses, and open challenges.)

~~~
wookietrader
Okay, I should have worded that differently. There is also a paper of
Salakhutdinov learning a kernel for Gaussian processes. That'd account for
that as well.

My point is (I did not really write that above) that deep learning does not
stand unchallenged in this domain. Its dominance is so far "only" apparent in
vision and audio classification tasks.

------
wookietrader
If an article says that Andrew Ng is "the man at the center of deep learning"
it's just not right. Geoffrey Hinton's and Yoshua Bengio's impact were at
least as high as his, if not much higher.

------
kinofcain
There is a very quick reference to the person who inspired him, Jeff Hawkins,
whose book is worth a read:

[http://www.amazon.com/On-Intelligence-Jeff-
Hawkins/dp/080507...](http://www.amazon.com/On-Intelligence-Jeff-
Hawkins/dp/0805078533/)

Edit: update link

~~~
netrus
Link does not load CSS for me, but this does:

[http://www.amazon.com/Intelligence-Jeff-
Hawkins/dp/B000GQLCV...](http://www.amazon.com/Intelligence-Jeff-
Hawkins/dp/B000GQLCVE)

~~~
kinofcain
Thanks, had stripped the tracking codes off the mobile link, but it doesn't
load right on desktop.

------
EthanHeilman
We should stop trying to claim every new method is "like the brain". We don't
have any clear understanding of how the brain works. One can be inspired by a
particular and likely wrong cognitive theory, but one can not say one is
building "machines that can process data in much the same way the brain does"
truthfully without a deeper, and currently unavailable, understanding of the
functioning of the human brain.

~~~
Jach
I feel the same way, that researchers should stop trying to mimic the brain,
but not because we don't understand the brain. While I think there are still
several decades before we'll be able to have mind uploads, I also think a lot
of people underestimate the quality of modern brain science. In any case, I
have the same reason as Dijkstra for why I think mimicking the brain isn't
that great an idea. In
[http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036...](http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html)
(really a great read to branch all sorts of thoughts off of) Dijkstra said,
"The effort of using machines to mimic the human mind has always struck me as
rather silly: I'd rather use them to mimic something better."

It's probably a harder problem, creating smarter-than-human intelligence on a
machine, but research isn't as constrained by laws and ethics (they don't have
to bemoan not being able to experiment with living human brains). I wish more
people were active in the area.

~~~
visarga
They are trying to find THE ONE ALGORITHM that solves all A.I. problems. The
brain has an implementation of it, but it is in wetware, hard to extract. Deep
learning makes some pretty good approximations of the visual areas, though.

~~~
EthanHeilman
You assume that human cognition has an algorithmic component. Maybe you are
right, but we still have a pretty shaky understanding of how Neurons work, let
alone how the brain works on a large scale. Who knows, lets investigate by
trying possibilities but lets understand that we are still in a position of
ignorance.

We have some probabilistic models of that successfully predict various future
states of the brain from past states or stimuli. This is not the same as
understanding it or even approximating it.

------
hvs
OT: His online Machine Learning class last year was great. He is the best
professor I've ever had, and explains things so clearly that you understand
them the first time. You are lucky if you ever get to work or study under him.

~~~
rzendacott
That's awesome to hear! I'm taking the Coursera class now, and it's been great
so far. It just started a couple weeks ago, so it definitely isn't too late to
join! <https://www.coursera.org/course/ml>

~~~
psbp
I wish Coursera followed the Udacity model. I always find out about these
classes after they're already weeks in progress or over.

~~~
pfg
You can star any Coursera class to receive notifications whenever new sessions
are announced.

Also, I believe it's still possible to join the current session (first
assignment was due this weekend, but you can turn it in late with just a 20%
penalty.)

~~~
visarga
I take new courses at any time, even if they have ended. Later on, when they
recycle, I can do them all over again with ease.

------
gavanwoolery
Hmm...I don't mean to be a skeptic, but I do not see any new theories here.
Neural networking has been around for a long time, as have an abundance of
theories and implementations around it...some people have gone so far as to
build an actual brain replica (a digital version of the bio/analog thing).
Neural networking is extremely powerful, but to be of any use, you need a
_lot_ of computing power. As it turns out, our brains are really good at
massively parallel tasks like image and motion processing; these things can be
done explicitly on a computer with some ease, but having a computer learn on
its own from scratch how to do them is not easy.

~~~
jbelanich
You're correct in that neural networks as a model have been around for a long
time. However, those networks were restricted to be shallow because
backpropogation didn't work well on networks with many hidden layers. Only
recently have researchers developed learning procedures that can learn these
deep architectures efficiently, using some clever unsupervised learning
techniques. And surprisingly, they are finding that these deep networks
perform remarkably well, beating the state of the art in a number of
benchmarks.

You are also right that you do need a lot of processing power to get neural
networks to work well. But that is changing rapidly. Hinton's convolutional
neural network has the state of the art in the ImageNet benchmark, yet was
trained using significantly less power than google brain. Regardless, you
don't need google scale computation to get deep networks to work well. The
point of google brain is to see how far one could push neural networks.

~~~
jcrites
> Only recently have researchers developed learning procedures that can learn
> these deep architectures efficiently, using some clever unsupervised
> learning techniques.

Would you mind naming some of these techniques, if you're familiar with them?
I'd like to take a deeper look.

~~~
visarga
Restricted Boltzmann Machines. <https://www.youtube.com/watch?v=AyzOUbkUf3M>

This video drives the point home, and is made by the author of this technique.

------
why-el
So I am a little confused. Where are we on the learning part of AI? As I
understand it, the current consensus is to throw as much data as you can at
your model (millions of cat pictures in this article's example) to make it
pick up patterns and yet still claim that we are closing in on how the brain
works? As far as I can tell no human brain would need that many pictures to
see a pattern. In fact, and this is probably more apparent in language, we
humans tend to work with _degenerate_ data and still end up with perfect
models.

~~~
piglop
You may not have seen millions of cats, but how many images have your brain
processed since you can see? A five years old brain has been trained on
billions of images (along with many other simultaneous inputs).

------
pilooch
It seems the relationship between the brain and deep learning has evolved in
such a way that the later can help with insights into how the former works.

In this regard, I thought I would mention the extraordinary simple and elegant
talk by G. Hinton last summer: <http://www.youtube.com/watch?v=DleXA5ADG78>

It starts from a simple and clever improvement to an existing deep learning
method and ends up with beautiful (and simple!) insights on why neurons are
using simple spikes to communicate.

------
aespinoza
I thought the man behind the Google Brain was Ray Kurzweil
(<http://www.wired.com/business/2013/04/kurzweil-google-ai/>).

~~~
drcross
Kurzweil is a hack. (Source: I've an Msc in Robotics and read all his books)

~~~
konstruktor
Please think about what a source is. This isn't reddit.

~~~
Cushman
Bullshit. There is nothing wrong with using personal experience for reference
so long as it is properly cited.

It beats "This guy is wrong (source: random blog post that I didn't really
read)", anyway.

Edit: To be clear, I don't love the Reddit snowclones, but there's nothing
wrong with the sentiment behind "I'm a scholar in this field, and I think this
guy is a hack."

~~~
konstruktor
So misapplying the term source and saying that one has a degree gives one
enough credibility as a scholar to turn a dismissive and colloquial statement
into a worthwhile contribution to a discussion about AI?

~~~
nostrademons
The point of sourcing your statements is to give the listener enough context
to judge for themselves what your credibility is. It's not a layperson's
responsibility to shut up, only to avoid misrepresenting themselves as an
expert. It's the listener's responsibility to judge the credibility of those
they listen to.

(Besides, my read of the comment that started this was that it's quite tongue-
in-cheek. He was basically saying "Don't trust this any more than any other
comment you read on the Internet.")

------
cscurmudgeon
Ah. The good old AI cycle.

Scientist: X can help us get full AI!

You: Why?

Scientist: Because of reason R.

You: But, reason R is a non sequitur...

More seriously, reasons similar to that for deep learning have been repeated
multiple times in AI with failure (e.g. Thinking Machines).

I would suggest that these folks remain calm and build something on the scale
of IBM's Watson using just deep learning..

------
niklaslogren
Very interesting article, it makes me hopeful.

This might be slightly off-topic, but I'll try it here anyway: can anyone
recommend any books/other learning resources for someone who wants to grasp
neural networks?

I'm a CS student who finds the idea behind them really exciting, but I'm not
sure where to get started.

------
yankoff
Great and inspiring professor. Taking his ML course on coursera and trying to
follow his talks.

