

What's wrong with convolutional nets? [video] - drewvolpe
http://techtv.mit.edu/collections/bcs/videos/30698-what-s-wrong-with-convolutional-nets

======
jfoutz
Short, massively simplified, summary,

Vast improvement in unsupervised learning.

Walks through biases humans have in object recognition. Explains new
structures to add on top of neural networks.

Basic backprop on MNIST with 30,000 labeled examples gives 1.7% error rate.
Hinton's approach gets the same results with 25 examples.

~~~
dchichkov
With 25 examples on which the systems asks the labels (and unknown number of
unlabeled examples).

... liked the progress on reverse computer graphics approach. A bit of
personal perspective here, from the data that I saw (and labeled!), while
working on hand gesture recognition project the reverse computer graphics felt
like the only approach that could bring human level gesture recognition from
2d data. And I was really exited at some point when Nvidea had started a
computer vision / gesture recognition group. It felt really right that a
photographic-level computer graphics GPU company would try to do reverse
computer graphics processor. Too bad it didn't go anywhere.

------
robg
One of the few speakers I saw as an academic and said to myself "This guy is a
time traveler. I need to really pay attention to what he's saying."

That was in 2001. A decade+ later and we're still trying to catch up.

~~~
TheCoreh
This precisely describes the feeling I got when watching the video. He
consistently provides explanations/examples that make complete sense in
hindsight but are pretty hard to come up with in the first place. I feel like
this video kinda deserves a "spoiler alert" of sorts because he just exposes
many of the things and it ends up looking so obvious afterwards. Not obvious
in a "yeah the brain certainly works like that" way, because we don't know
that yet, but obvious in a "I really wouldn't be surprised if it worked like
that" way.

------
TrainedMonkey
Video buffers every few seconds for me, are there any mirrors?

Edit: found download link after getting some caffeine
[http://techtv.mit.edu/videos/30698-what-s-wrong-with-
convolu...](http://techtv.mit.edu/videos/30698-what-s-wrong-with-
convolutional-nets/download.source)

Edit 2: Also, there is a class on Coursera where he shares much knowledge and
intuition about neural networks. I personally took it and it was excellent.
Unfortunately there are no repeating sessions, but even archived material is a
goldmine.
[https://class.coursera.org/neuralnets-2012-001](https://class.coursera.org/neuralnets-2012-001)

~~~
tripzilch
I just pointed youtube-dl at the original URL and it detected and downloaded
the embedded .mp4 just fine.

------
discardorama
If I'm understanding some of the criticism, it is basically that we don't
think "that's how our brain does it" (when it comes to convnets, pooling,
etc.). But this begs the question: do we _have_ to duplicate what goes on in
the brain? For example: we really don't know how the brain does
multiplication; and if I were to guess, I'd say it has nothing to do with
binary arithmetic and shift registers. But yet we've been able to teach a
computer how to do it, using an algorithm that has (in all probability) no
relation to how our brains work. So maybe convnets, pooling and other hacks
work, just not how the brain works?

------
akyu
A lot of these sound like ideas that Jeff Hawkins has been pushing with his
Cortical Learning Algorithm

~~~
trentonstrong
It's interesting since Hinton's approach seems to be more inspired by the
computational geometry aspects of vision (reverse computer graphics, invariant
representations) and working backwards towards the neural superstructures.

Hawkins' seems to be inspired by the biology itself, working from the "common
computational substrate" hypothesis up to the cortical units required.

They both seem to meet in the middle at the need to figure out invariant
representations for the intermediate features presented to each level of the
hierarchy.

I do wonder what an argument between those two would look like concerning the
applicability of back-propagation, which I remember Hawkins' deriding as
totally artificial compared to the feedback structure of the actual neocortex.

Anyone more up to date on the state of the argument regarding that?

~~~
MrQuincle
Backpropagation is just calculating the gradient of a multilayer perceptron.
You do not even have to calculate it, you can use autodiff for it:
[http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autod...](http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autodiff_nnets.pdf)
(pdf).

They both seem to neglect time (internal dynamics). I haven't seen either of
them coming up with a model as that from Izhikevich with polychronization:
[http://www.izhikevich.org/publications/spnet.htm](http://www.izhikevich.org/publications/spnet.htm).
If we would be able to make one of such ideas computationally useful, things
would become really interesting.

~~~
trentonstrong
Interesting, thanks for the links!

I've done your typical ANN 101 training in the past so have a good mental
model for back-propagation. Modelling the actual nonlinear dynamics of
"realistic" neural networks seems like an obvious path of research but I know
how daunting it is. It seems like every tiny bit we can push forward our tools
for understanding complex non-linear systems should pay large dividends across
so many different computational fields (fluid dynamics, QFT, economics, ...,
everything?)

I'll have to read Izhikevich's paper, seems like a unique line of research.

~~~
MrQuincle
Absolutely nice to check his work. He founded a startup, Brain Corporation
([http://www.braincorporation.com/](http://www.braincorporation.com/))
recently.

Coincidence detection where delays are playing a functional role is one of the
things that I find interesting (as well as the fact that there are more
polychronous groups than neurons). The other thing is the emergence of gamma
waves. I would be surprised if these do also not have some functional role
(although it might be just as well the humming of our biological processor).
:-)

I wish I was brave enough to start experimenting with different neural
networks. For now I am on the roll "Bayesianfying" everything I encounter.
Even the Hough transform that Hinton is so fond of in this talk. :-)

------
jacobn
Does anyone have a link to the presentation itself (e.g. ppt/pdf)?

