
Link Between Deep Neural Networks and the Nature of the Universe - taeric
https://www.technologyreview.com/s/602344/the-extraordinary-link-between-deep-neural-networks-and-the-nature-of-the-universe/
======
rdlecler1
It's not that no one understands, it's that journalists don't know that
their's research that has addressed this question. The problem with
intuitively understanding neural networks is that most of the weighted edges
are spurious. Meaning you can remove most of them (WiJ -> 0) without
negatively affecting the mapping function solution. Failing to do this we
stare at a fully connected neural network which gives us no clues why it
works. However , once you get rid of these spurious interactions (through
perturbation analysis), then This will reveal the underlying circuit diagram
for the network.

Often these threshold integration neurons are just approximating a simpler
Boolean function so it really comes down to digital logic.

Techniques for pruning networks can be found here:

[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/)

~~~
smaddox
Pruning works for understanding sparse networks, but not for holographic
networks, which can encode a much higher information density. Understanding a
holographic network requires a large amount of effort, similar to determining
the content of an optical holograph. You can try to develop a model, but if
the information density is very high, it's no faster than observing the
response to various inputs.

The problem becomes even more difficult if there is no clear distinctions
between layers in the network, as in the brain.

~~~
rdlecler1
A network should be no more complex than it needs to be. If you're not pruning
_spurious_ interactions then you're implicitly assuming that each interaction
is information bearing. If you're not pruning spurious interactions from
holographic networks then you're assuming that they're all functionally
relevant. I promise you they're not.

You're also assuming that holographic networks are somehow necessary for most
of the tasks we require. The human brain, while having billions of neurons is
still quite sparsely connected.

------
alehander42
I think the point is that all the patterns we deal with are a result of physic
laws, so that somehow limits the solution space

~~~
akvadrako
That seems to be concise summary of the articles point, but my first reaction
is that it's a superficial distinction.

One of the researches is Max Tegmark, author of Our Mathematical Universe. In
short, he suggests every theory becomes it's own universe; physical laws are
representations of patterns in some kind of calculation or proof.

As he explains, although every possible theory is a universe, "most" complex
theories can be reduced to simple ones. So it seems plausible that those
simple patterns are more common and more likely to be found in a random
(mathematical or physical) universe.

This means that neural networks are likely to work with those common patterns.
One might say it because that's closer to the nature of physical laws. But
more directly, that's because of the nature of mathematical laws.

------
mamon
I think that theoretical physics is an example of confusing map with the
territory. Just because you can work out some mathematical mumbo-jumbo does
not necessarily mean that it has any connection to reality.

This applies especially well to string theory which is just layers of
pointless mathematics, each layer trying to make up for obvious deficiencies
of lower layers.

------
Houshalter
I wrote a better explanation for why neural nets work here:
[http://houshalter.tumblr.com/post/120134087595/approximating...](http://houshalter.tumblr.com/post/120134087595/approximating-
solomonoff-induction)

The answer is that neural networks are a somewhat crude approximation of
Solomonoff induction. Solomonoff induction is an ideal, perfect machine
learning method. It only assumes that the universe is a random computer
program, and tries to infer which one.

~~~
faragon
Also interesting is the relation between Solomonoff induction and Kolmogorov
Complexity.

------
ankurdhama
Take any two fields, keep abstracting away stuff and you will reach a stage
where you find a link between the two fields.

------
adityab
You should watch Max Tegmark's talk "Connections between Physics and Deep
Learning" if this interests you [1].

Additionally, a paper that has everyone excited about deep connections between
the mathematical analysis of physical systems and the hierarchical feature
learning paradigm speaks of the connection in terms of the Renormalization
Group [2].

Regardless of it's practical utility, the philosophical implications do tickle
the intellect. On a dreamy note, I wonder if it would be possible to draw
Category-theoretic parallels between some physical theories and statistical
learning theory. There is so much to learn, and I am trying my best to teach
myself (on the side) the mathematics that they don't teach in my CS grad
school. So much to learn, such little time. :)

[1] [https://youtu.be/5MdSE-N0bxs](https://youtu.be/5MdSE-N0bxs)

[2] [http://arxiv.org/abs/1410.3831](http://arxiv.org/abs/1410.3831)

------
divan
> Despite the huge success of deep neural networks, nobody is quite sure how
> they achieve their success.

Is that true? After reading Jeff Hawkings - "On intelligence" book (2004), for
me it's pretty much clear why they are so successful on that set of tasks.
Despite of HTM model built by Jeff being kind of different from DNN, the idea
and origin of brain and consciousness described pretty well. All modern
advances in neuroscience just prove how strong the theory in this book is.

~~~
bbctol
Jeff Hawkins has some pretty heterodox theories of intelligence, many of which
are not backed up by modern neuroscience. Furthermore, as neuroscience
advances, the similarities between so-called artificial neural networks and
actual neurons are rapidly receding. Much of the current success of
convolutional neural networks is (loosely) inspired by a particular part of
the anatomy of the eye, but non-visual processing and abstract reasoning seems
to take place under much more complicated patterns that ANNs may not be able
to replicate as easily.

------
Tycho
Going a bit further, Chris Langan's _CTMU_ characterised the universe itself
as a sort of back-propagation algorithm with a global utility function.

------
faragon
Neural networks are trained with information from the Universe. So, in my
opinion, it should not be shocking at all that neural networks store some
projection/model/echo of the universe (e.g. physical laws, maths, etc.).

