
Information Theory for Intelligent People (2018) [pdf] - oska
http://tuvalu.santafe.edu/~simon/it.pdf
======
mac01021
The original exposition by Claude Shannon, though long, is one of the most
readable mathematical publications I've ever encountered.

It's hard to imagine anyone improving on it.

[http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...](http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)

------
Vaslo
This is the opposite of pdf articles on HN. Many are titled with something
like "beginners" or "for everyone", then I open up a Latex formatted document
that is impenetrable from the first page using math symbols I have never seen
before. this is the opposite, much easier to understand and not loaded up with
mathematics symbol vomit all over the page.

------
jonmc12
For another perspective of "the study of information in the social world",
check out "Variational ecology and the physics of sentient systems"
([https://www.sciencedirect.com/science/article/pii/S157106451...](https://www.sciencedirect.com/science/article/pii/S157106451930003X)).
Also based on information theory, released just a few months after op paper.

------
jbhatab
I'm a pretty confused on how this all ties together. Are these these deep
dives on examples on information theory? Would love some perspective that
knows more on the topic.

~~~
boyobo
It's an introduction to the fundamental concepts of information theory
(entropy, KL divergence, conditional entropy), using 20 questions as an
ongoing source of example applications and interpretations of these concepts.
The author also sprinkles in various interpretations coming from their own
research focus (the social sciences).

Definitely not a deep dive. The details can probably be found by chasing down
the citations.

~~~
jbhatab
Awesome thanks so much

------
simondedeo
Super-pleased to see the attention to this little piece. It gets updated from
time to time, and it's always useful to see comments. I appreciate it.

Some responses:

1\. hos234 asks "Why do they use the word Release? Release of entropy suddenly
drops into the picture when talking abt ice to water. Why not just say
info/entropy/uncertainty of the system is increasing or decreasing?"

It can be useful (or, at the very least, fun) to think of uncertainty as a
fluid that moves through a physical system. Imagine an air-conditioner, for
example; it is doing work, but not changing the overall energy of the system
(at least in approximation and if it is working very slowly). It's just
separating out particles.

The more advanced version is Maxwell's Demon; Rolf Landauer at IBM did the
full analysis of this system, which you can think of as creating useful energy
by "consuming" a pristine hard-drive (or, equivalently, dumping entropy into
it.)
[http://www.w2agz.com/Library/Limits%20of%20Computation/Landa...](http://www.w2agz.com/Library/Limits%20of%20Computation/Landauer%20Article,%20Physics%20Today%2044,%205,%2023%20\(1991\).pdf)
or
[https://www.sciencedirect.com/science/article/pii/0375960196...](https://www.sciencedirect.com/science/article/pii/0375960196004537)

2\. I called it "for intelligent people" because I wanted to emphasize that
just because you don't know what information theory is, or are not
particularly mathy, it doesn't mean the author (me) thinks you're dumb. The
hope was to counteract the usual physics-style arrogance that xkcd captures in
"why does your field need a journal?"
[https://xkcd.com/793/](https://xkcd.com/793/) And yes, per jldugger, it was a
play on the "for Dummies" series.

Ironically for me, the humanities people that it was aimed at often took it
amiss. Dan Dennett told me as much. throwawayjava thank you for the
suggestion; I'll use it in a footnote in the next iteration!

3\. if you liked "information theory for intelligent people", you'll LOVE
"Bayesian reasoning for intelligent people", the shocking prequel featuring
characters that will ruin everything for you and plus be mildly racist?
[http://santafe.edu/~simon/br.pdf](http://santafe.edu/~simon/br.pdf)

~~~
boyobo
_Ironically for me, the humanities people that it was aimed at often took it
amiss. Dan Dennett told me as much. throwawayjava thank you for the
suggestion; I 'll use it in a footnote in the next iteration!_

Yes, I thought this article was a very well written introduction, but I held
back on sharing this link with some non-math friends because I didn't want
them to feel dumb if they find it difficult to get through.

------
hos234
Why do they use the word Release? Release of entropy suddenly drops into the
picture when talking abt ice to water. Why not just say
info/entropy/uncertainty of the system is increasing or decreasing?

This subject is new to me but I like the 20 questions analogy.

Thinking of probability distribution of N words as a measure of info in
someone's mind is interesting.

Disciplined minds that have certain concepts on their finger tips clearly have
spent time and effort producing that distribution.

~~~
n4r9
True, it does seem a bit weird to use "release" in reference to entropy. I've
googled around a bit and can't find a similar phrasing anywhere. A lot of
places talk about "energy" being released into the surroundings, and this
article talks about using the entropy of the surroundings to predict whether
or not ice will melt:
[http://www.4college.co.uk/a/O/entsurr.php](http://www.4college.co.uk/a/O/entsurr.php)
.

------
Ericson2314
Why the title?

~~~
jldugger
You know the book series 'Computers for Dummies?' It's a play on that. See
also Zytrax's 'X for Rocket Scientists' which covers complicated stuff like
LDAP and DNS and explains _why_ they are the way they are.

~~~
dredmorbius
Joe Celko's _SQL for Smarties_ ([https://www.oreilly.com/library/view/joe-
celkos-sql/97801280...](https://www.oreilly.com/library/view/joe-celkos-
sql/9780128007617/)) and "Learn X the Hard Way" (Python, Java, C/C++, etc.)
also play on the "for idiots/dummies" motif.

------
shitgoose
try this:

[https://colah.github.io/posts/2015-09-Visual-
Information/](https://colah.github.io/posts/2015-09-Visual-Information/)

his blog is pretty cool. he is original.

~~~
rramadass
Nice Article!

Much reading to do on his blog :-)

------
oarabbus_
Hmm, Information Theory has always done an excellent job of making me feel
non-intelligent, so not sure how applicable this PDF is to me.

