Wiener made important contributions to mathematics but not to information theory. He wrote a book saying that entropy is related to "information" and is maximized for a Gaussian. That's about his involvement. The paper attached goes way beyond that.
This "surprisal" can be parametrized by relative entropy (Kullback-Liebler divergence), which reduces to the Shannon entropy formula only when the observer has the correct anticipated distribution.
> With Shannon’s startling ideas on information, it was one of the rare moments in history, an academic would later point out, “where somebody founded a field, stated all the major results, and proved most of them all pretty much at once.”
It is attributed to a documentary on Claude Shannon: https://www.youtube.com/watch?v=z2Whj_nL-x8
Another cool fact: Shannon’s master thesis is most likely the most influential one of all time: in it, he linked Boolean algebra to electrical circuits with switches, essentially inventing digital circuit theory.
I highly recommend William Poundstone's book, "Fortune's Formula" as a biography of those ideas - it's almost as good as any Michael Lewis book on the subject would be.
It was quite a joy and I highly recommend to anyone interested in these sorts of bios.
I never fail to suppress a chuckle when I read that.
2016, 53 comments:
2017, 11 comments:
In trying to connect Shannon entropy to thermodynamic entropy, I always get stuck on the fact that you need to have a defined alphabet or symbol set.
Their are more exotic mechanisms to define entropy: https://en.wikipedia.org/wiki/Parastatistics
I don't see how to square those manifest physical effects with the immateriality of Shannon information entropy.
In the Shannon formulation, there isn't a notion of "too much information" -- it is all symbols.
But in reality, too many photons can burn our eyes or a denial of service attack can break a website. Too much information is a reality --because we are thermodynamic entities.
I don't see how to understand that with Shannon.
Here is a great, short overview of the topic => https://youtu.be/_PG-jJKB_do
There is no entropy regarding question of whether the Jade is an attractive presenter.
Please don't do this here. It just comes across as super-crass.
I find MacKay's explanations very clear. Sadly, he passed away far too young.