
The Strange Loop in Deep Learning - ceperez
https://medium.com/intuitionmachine/the-strange-loop-in-deep-learning-38aa7caf6d7d
======
famil
Jürgen Schmidhuber had an interesting comment on the Strange Loop in his AMA:
[https://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_a...](https://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_am_j%C3%BCrgen_schmidhuber_ama/)

As we interact with the world to achieve goals, we are constructing internal
models of the world, predicting and thus partially compressing the data
history we are observing. If the predictor/compressor is a biological or
artificial recurrent neural network (RNN), it will automatically create
feature hierarchies, lower level neurons corresponding to simple feature
detectors similar to those found in human brains, higher layer neurons
typically corresponding to more abstract features, but fine-grained where
necessary. Like any good compressor, the RNN will learn to identify shared
regularities among different already existing internal data structures, and
generate prototype encodings (across neuron populations) or symbols for
frequently occurring observation sub-sequences, to shrink the storage space
needed for the whole (we see this in our artificial RNNs all the time). Self-
symbols may be viewed as a by-product of this, since there is one thing that
is involved in all actions and sensory inputs of the agent, namely, the agent
itself. To efficiently encode the entire data history through predictive
coding, it will profit from creating some sort of internal prototype symbol or
code (e. g. a neural activity pattern) representing itself [1,2]. Whenever
this representation becomes activated above a certain threshold, say, by
activating the corresponding neurons through new incoming sensory inputs or an
internal ‘search light’ or otherwise, the agent could be called self-aware. No
need to see this as a mysterious process — it is just a natural by-product of
partially compressing the observation history by efficiently encoding frequent
observations.

