
Better Intuition for Information Theory - hardmaru
https://www.blackhc.net/blog/2019/better-intuition-for-information-theory/
======
adriantam
Yeung (author of the article mentioned) was my teacher in school. He wrote a
book more than a decade ago with a Venn diagram of four sets on the cover

[https://www.amazon.com/First-Course-Information-Theory-
Techn...](https://www.amazon.com/First-Course-Information-Theory-Technology-
ebook/dp/B000TNLRVY/)

I remember he draw such diagram almost every lesson to explain the
mathematical structure of various information theory topics.

~~~
0-_-0
I just went down the rabbit hole of Venn diagrams, and emerged with one for 7
sets:

[https://i.stack.imgur.com/kImwq.png](https://i.stack.imgur.com/kImwq.png)

~~~
carapace
That's truly awesome. Cheers!

Check out Lewis Carol's Logic Game:

[https://www.cut-the-knot.org/LewisCarroll/tridiagram.shtml](https://www.cut-
the-knot.org/LewisCarroll/tridiagram.shtml)

------
knzhou
This is nice, but I could've sworn I saw the exact same diagrams in all the
old-school information theory books I originally learned it from. I've drawn
the I-diagrams the author gives in my notes quite a few times. I'm guessing
the point of the paper is that it goes further than that?

~~~
kthielen
The paper is from 1991, when was your textbook written?

~~~
mturmon
The first place I saw these diagrams is in the first edition of Tom Cover and
Joy Thomas's excellent textbook - copyright 1991.

But the Venn diagrams are presented in their book as mnemonic tools, and not
in terms of entropies of an underlying "I-measure" as in the article. (Of
course, the book is an introduction, not in a theorem/proof format.) So you're
not sure how far the notions can be extended.

The world of information theory was small then, and the author of the paper
would have been well-known to both Cover and Thomas.

