
Exact mapping between Variational Renormalization Group and Deep Learning (2014) - _qc3o
https://arxiv.org/abs/1410.3831
======
eximius
Can anyone on here can elaborate whether this a) makes sense at all and b)
what its implications are for physics and deep learning.

~~~
charleshmartin
See my blog: [https://charlesmartin14.wordpress.com/2015/04/01/why-deep-
le...](https://charlesmartin14.wordpress.com/2015/04/01/why-deep-learning-
works-ii-the-renormalization-group/)

I try to highlight the paper and some of the history and relevance, in a
general way, but also hitting the math hard.

the more general idea, which I am still formulating, is that DL systems are
very different from traditional ML (ala VC theory)

In traditional ML systems, we tune the regularizer to optimize the capacity of
the learner

In deep learning, we would optimize both the capacity (entropy) of the
learner, and the optimization problem (energy function)

This is also what happens in the stat mech of protein folding, where the
energy is optimized, even when we are at minimum capacity. This gives rise to
a funneled energy landscape

[https://charlesmartin14.wordpress.com/2015/03/25/why-does-
de...](https://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-
learning-work/)

Similar behavior is seen generally in stat mech near a critical point, and
this is why the RG analogy is relevant for me.

We should be able to see this behavior if we simply plot the entropy vs energy
of say an RBM. Im not entirely sure yet how general this is, but it works for
MNIST.

I discuss (some of) this in more detail in a video from a talk I gave this
summer at MMDS
[https://www.youtube.com/watch?v=kIbKHIPbxiU](https://www.youtube.com/watch?v=kIbKHIPbxiU)

~~~
dldl
Is it related to the predatory/prey population problems we did in Linear
Algebra?

Something like this:
[http://www.math.umd.edu/~jmr/246/predprey.html](http://www.math.umd.edu/~jmr/246/predprey.html)

~~~
mlechha
Interesting. I'm curious as to why you make that connection? What reminded you
of predator prey systems?

------
MrQuincle
What's the decimation operator?

Authors claim an exact mapping, but fail to define a few of the essential
elements of any renormalization group approach.

