
Information Theory with TensorFlow - behnamoh
https://www.adhiraiyan.org/deeplearning/03.00-Probability-and-Information-Theory#13
======
unlinked_dll
I feel there is some great unification of Information Theory, classical signal
processing, and controls with machine learning. Fundamentally they are
different angles of applying the concepts of dynamical systems to design
widgets that do something useful. Things like this quote from the top:

> Information theory is a branch of applied mathematics that revolves around
> quantifying how much information is present in a signal.

greatly underappreciate the value of the theory as it applies to the theory of
things like ML, and one day I think we'll have a unifying theory that enables
engineers to design systems for ML with little ad-hoc methods and more
theoretical bases as we design filters and controls today. The issue is the
nonlinearity of the networks, but I think we'll find a way there through
category theory and topologies.

~~~
ajtulloch
This is very close to the thesis of Mackay’s Information Theory, Learning, and
Inference textbook:
[https://www.inference.org.uk/itprnn/book.pdf](https://www.inference.org.uk/itprnn/book.pdf)

~~~
ignoramous
Video lectures:
[http://videolectures.net/david_mackay/](http://videolectures.net/david_mackay/)

Also see:
[https://news.ycombinator.com/item?id=11500221](https://news.ycombinator.com/item?id=11500221)

------
xvilka
Information Theory with Coq[1]. Formal approach.

[1] [https://github.com/affeldt-aist/infotheo](https://github.com/affeldt-
aist/infotheo)

