
Why does unsupervised deep learning work? - MrQuincle
https://arxiv.org/abs/1412.6621
======
MrQuincle
To summarize:

\+ An autoencoder is a stabilizer of the input f: it maps it to itself.

\+ Imagine that the space of autoencoders forms a group.

\+ Learning stops as soon as a stabilizer is found.

\+ If the search is a Markov chain, the bigger the stabilizer the sooner it
will be hit.

\+ The group structure implies that this big stabilizer corresponds to a small
orbit.

\+ The orbit of an element x in X is the set of elements x can be moved to by
the elements of G (the group).

\+ The stabilizer is the set of group actions that map x to itself.

\+ Blog post explaining fundamentals:
[https://gowers.wordpress.com/2011/11/09/group-actions-ii-
the...](https://gowers.wordpress.com/2011/11/09/group-actions-ii-the-orbit-
stabilizer-theorem/)

\+ Reconstruction in deep NNs is often guided by an l2 distance. If there are
competing feature sets, gradient descent moves the configuration to one of the
stabilizers.

\- Note. Is this indeed the case? Is there a WTA mechanism at play? Is this
preferable?

\- Note. In this view a NN is a MCMC that is stopped prematurely.

\+ The probability that a network "discovers" a stabilizer for the signals f_i
depends on the volume of the stabilizer.

\- Would we be able to construct a PDF by running an MCMC till we are in a
high probability region? Or does this only work if we're interested in modes?

\+ A neural network operation is not a group action but can be seen as a
"shadow group" in some auxiliary space (theorem 4.1, 4.2).

\+ The size of the stabilizers is preserved in the original space (theorem
4.3).

\+ Next levels exhibit again symmetry, generalized edges lead to figures like
trapezoids, triangles, and butterflies.

Most important take-away: representations that exhibit a lot of symmetries are
the ones that are easiest/fastest to find.

~~~
Ericson2314
This could be the least dismissive and anti-intellectual tl;dr I've seen here.

------
Ericson2314
Will deep learning soon make the jump from alchemy to chemistry? Will abstract
algebra suddenly attract all the business-brogrammers? Stay tuned!

~~~
Ericson2314
To be clear, this paper looks great. My concern is the poeple pushing deep
learning before the theory is worked out—it's an AI winter waiting to happen.

