
A 'Brief' History of Neural Nets and Deep Learning - andreyk
http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/
======
dahart
Excellent article, thanks for the contribution.

Perhaps some independent validation, but I was coincidentally having this
conversation the other day with a relatively well known computer vision
researcher, about why it seems like the idea of neural nets has floundered for
decades and suddenly it's the hot topic, and we're seeing massively improved
results.

His answers, summarized, are that:

1- Big data is making possible the kind of training we could never do before.

2- Having big data & big compute has made some training breakthroughs that
allowed the depth to increase dramatically. The number of layers was
implicitly limited until recently because anything deep couldn't be
practically trained.

3- The activation function has very commonly in the past been an S-curve, and
some of the newer better results are using a linear function that is clamped
on the low end at zero, but not clamped on top.

All really interesting to me. This is making me want to implement and play
with neural nets!

Of course, now the big question: if we have a neural net big enough, and it
works, can we simulate a human brain? (Apparently, according to my AI
researcher friend, we're not there yet with the foundational building blocks.
He mentioned researchers have tried simulating a life-form known to have a
small number of neurons, like a thousand, and they can't get it to work yet.)

~~~
dwiel
I agree with everything but the last. Simulating a human brain is perhaps
necessary for consciousness uploading, but not for super human ai. We have
been able to travel faster by machine than foot for a long time, in spite of
not being able to build a robot that can simulate human running.

~~~
dahart
> Simulating a human brain is perhaps necessary for consciousness uploading,
> but not for super human ai.

I'm not suggesting otherwise, and what you said might be true, I don't know.
But honest question: how can we know it's true before we can do either? How
can we suggest neural nets are a good building block of super human AI, if we
can't verify we can do regular human thinking, or sub-human thinking?

Because the neural net is modeled after (simplified) brain neurons in the
first place, it seems reasonable to suggest that if we can't simulate a simple
brain, then we can't validate that the model is correct, right? It might be
very productive, and it might 'work' in some sense, but we don't know whether
it can truly act as the lego brick of brain building material until we can
build a functional brain.

Your observation is true, and worth considering. Machines can travel faster
than humans, _but_ that's different in part because we didn't start trying to
simulate the human foot, right? The computational equivalent would be that
machines can multiply a lot of numbers much faster than a human. That is
super-human calculation, and it is mechanical thinking, of a sort, but most
people wouldn't say it counts as "AI", perhaps the same way that most people
wouldn't say a car counts as human running, even if it is faster.

Neural nets are being used to classify images, find objects, identify people's
faces in a crowd or in difficult to see situations. But, they currently can't
tell you to stop classifying images because you're asking the wrong question,
or interrupt you to say you're looking great today.

~~~
argonaut
Before we get wildly over-optimistic, as every HN thread on AI becomes,
realize that neural networks have basically nothing to do with actual
biological neurons. There are a bajillion things neurons do that neural
networks do not do. And there are similarly many things that are a part of
neural networks, that biological neurons do not do. In addition to the fact
that neuroscientists still do not have a clear understanding of biological
neurons: you cannot expect to reverse-engineer something you barely understand
yourself.

~~~
andreyk
This is not entirely true, as there are spiking neuron nets and neuromorphic
chips (like IBM's TrueNorth) that emulate what we have so far understood about
how neurons operate. But it's true that even these are simplified models and
as far as I know cognitive science has not explained the higher level
operation of the brain. In general I think breaking down how modern deep
learning works makes it seem unlikely that it will lead to true AI without
additional insights, so I am unsure why very intelligent people such as Elon
Musk make it seem like an imminent threat - not to mention the idea is
predicated on continued exponential increases in computing power. But the
stuff going on in AI right now concerning image comprehension and question
answering is fairly impressive, so I think it is feasible we will have
impressive AI assistants and the like within a decade.

~~~
pedrosorio
"This is not entirely true, as there are spiking neuron nets and neuromorphic
chips (like IBM's TrueNorth) that emulate what we have so far understood about
how neurons operate."

How is this related to the neural networks we are discussing? (deep ANNs?) As
far as I know, the ANN topic in machine learning, apart from its origins, is
completely unrelated to the simulation of biological neural network models:
[https://en.wikipedia.org/wiki/Nervous_system_network_models](https://en.wikipedia.org/wiki/Nervous_system_network_models)

They are often confused because of the name and because ANNs did begin as
extremely simplified models of biological neural networks, but the machine
learning concept that is setting all the records in vision / speech
recognition serves no purpose as a model of a biological neural network.

~~~
nl
_the machine learning concept that is setting all the records in vision /
speech recognition serves no purpose as a model of a biological neural
network._

I think you are overstating this. There are definite links between the two,
even if ANNs end up different because of the tools we have available to us.

CNNs were explicitly designed to mimic the behavior of the visual cortex.

Most of Geoff Hinton's career has been built around thinking very hard about
biological computation.

One of the major criticisms of back-propegation in neural networks is that it
is biologicaly implausible.

[https://en.m.wikipedia.org/wiki/Convolutional_neural_network...](https://en.m.wikipedia.org/wiki/Convolutional_neural_network#History)

------
mturmon
Upvoted -- what a nice and detailed history.

Thanks for linking to the old NYT article on Frank Rosenblatt's work. One can
see how researchers of the time were irked by delirious press releases when
the credit-assignment problem for multilayer nets had not been addressed.

(We managed to mostly address the credit-assignment problem for multilayer
nets...but the delirious press release problem remains unsolved.)

Incidentally, it's "Seymour Papert", not "Paper" (appears twice).

~~~
MrQuincle
This is too long ago for me to know what actually happened. Both Rosenblatt as
well as Minsky and Papert knew that networks could perfectly learn the XOR
function. See
[https://en.wikipedia.org/wiki/Perceptrons_(book)](https://en.wikipedia.org/wiki/Perceptrons_\(book\))
and also stated in this article.

I knew there was a fight between Minsky and Grossberg at that time. Perhaps
there have been other reasons that are not so well known that led to an AI
winter. Have these winters ever be quantified though?

For the article, you can find it in here:
[http://msrvideo.vo.msecnd.net/rmcvideos/258318/dl/258318.pdf](http://msrvideo.vo.msecnd.net/rmcvideos/258318/dl/258318.pdf)

------
FreedomToCreate
Great read. Its incredible what we, as a species have been able to achieve in
the last 2 centuries. I feel like we are where scientists were with computers
in the 1950s. We are starting to see the big picture, but its wide application
is still decades away.

------
ausjke
Great article, took an neural network course while I was doing my graduate
study long time ago and it might be time to resume that subject. the NN
training then took a long time to be impractical for real use and now it
should be much faster.

------
lovelearning
Author's done an excellent job of explaining what the problems were at every
stage and how NNs evolved to solve them. Learnt a lot from this series.

------
BasDirks
This is actually a very important chapter in human development. And it will
pass in the blink of an eye. Works like these are good, important.

------
signa11
just to re-iterate, the reference text #13 I.e. parallel distributed
processing (vol. 1&2) are also an excellent introduction to the field, from
its infancy.

contains a collection of papers by nn luminaries including rumelhart, Hinton
etc. Very highly recommended.

