
A Brief Overview of Deep Learning - tim_sw
http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html?m=1
======
a-priori
_This is not scientific fact since it is conceivable that real neurons are
much more powerful than artificial neurons, but real neurons may also turn out
to be much less powerful than artificial neurons. In any event, the above is
certainly a plausible hypothesis._

Biological neurons are indeed more powerful than (most) artificial neural
network models because these models discard important characteristics of
biological neurons:

* Spiking: Most artificial models are 'rate-based', where they gloss over the spiking behaviour of neurons by only modelling the firing rate. This discards all the various kinds of spiking behaviours (intrinsically spiking, resonators, bursting, etc.) as well as the relative timing of spikes. The relative timing is the basis for spike-timing dependent plasticity (STDP), which enables Hebbian learning and long-term potentiation -- two of the ways that networks learn to wire themselves together and process information.

* Conduction delays: Biological neural networks have a delay between when a spike is generated at the axon hillock and when it arrives at the postsynaptic neuron's dendritic arbour. This delay acts as a kind of like delay line memory in computers, where information can be 'stored' in-transit for short periods of time (in the ballpark of 0.5-40ms). And because different axons have different delays, information can be integrated over time by having one axon with a short delay and one with a long delay both end up at the same postsynaptic neuron.

~~~
mabbo
But on a computational power level, does that actually make them more
powerful?

What I mean is that Finite state machines are less powerful computationally
than context free grammars. A FSM cannot compute certain things that a CFG
can. Further, a CFG can't compute certain things that a Turing machine can.
But we _do_ know that Neural networks like the ones being used for Deep
Learning can compute anything a Turing machine can, and anything a Turing
machine can compute, so can the NN. They're equivalent.

So the real question is this: do those features (spiking, conduction delays)
actually make biological neural networks capable of computing something that
Turing Machines and Artificial Neural Networks cannot?

I hypothesize the answer is "no". A Turing machine could simulate any of those
features you've mentioned, and therefore an ANN could also simulate them. (But
I would _love_ to be wrong about it, that would be amazing if human minds
could do something that no machine would ever be capable of!)

~~~
Houshalter
Neural networks are universal function approximators, not Turing machines.
They can _theoretically_ learn any series of "if...then..." functions with
enough neurons. But there are a lot of functions they can't represent very
efficiently or without absurdly large numbers of neurons and training data.

~~~
a-priori
Yes, but computation can be performed with a function approximator where each
iteration is a function `f :: (state, input) -> (state', output)`. This is the
basis of an architecture called a 'neural turing machine'
([http://arxiv.org/abs/1410.5401](http://arxiv.org/abs/1410.5401)) and it is,
indeed, Turing complete and can be trained through standard neural network
training algorithms to perform arbitrary computations.

------
m0g
Biological analogies are too often misleading and confusing when talking about
deep learning[1]. We currently have very little knowledge of the way the brain
works and most analogies are only wild assumptions. The ones contained in this
article are blunt and based on strictly nothing but the author's feelings.
Please read with care.

[1]: [http://spectrum.ieee.org/robotics/artificial-
intelligence/ma...](http://spectrum.ieee.org/robotics/artificial-
intelligence/machinelearning-maestro-michael-jordan-on-the-delusions-of-big-
data-and-other-huge-engineering-efforts)

~~~
nl
The author's "feelings" are useful when they were one of the authors of
AlexNet[1]. A 10% improvement on ImageNet[2] makes one think he might know a
little about the subject.

[1]
[http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf](http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf)

[2] [http://www.image-
net.org/challenges/LSVRC/2012/results.html](http://www.image-
net.org/challenges/LSVRC/2012/results.html) (Look for SuperVision)

~~~
m0g
Well, I'm talking specifically about the biology analogies, and no, being good
at ML doesn't mean you know anything about the brain.

------
ot
Previous discussion:
[https://news.ycombinator.com/item?id=8888485](https://news.ycombinator.com/item?id=8888485)

------
postitnotecode
The author mentioned: _" a DNN with 2 hidden layer and a modest number of
units can sort N N-bit numbers"_, does anyone have a reference to this result?

------
pagnotta
I remember reading an article saying that the memory storage in the brain may
be at molecular level, making the brain's storage capacity really huge.
Anyway, there are much more things going on in the brain than just deep neural
networks.

