
A brief introduction to the beauty of Information Theory - justinucd
https://notamonadtutorial.com/a-brief-introduction-to-the-beauty-of-information-theory-8357f5b6a355
======
dentalperson
Shameless plug: I'm trying to start an online reading group for "Elements of
Information Theory" by Cover and Thomas.

If you're interested, please join
([https://www.reddit.com/r/mathreadinggroup/](https://www.reddit.com/r/mathreadinggroup/))
and the discord ([https://discord.gg/D9FjZXs](https://discord.gg/D9FjZXs)).

------
mikorym
This is an interesting topic and is closely related to what I am working on at
the moment.

One interesting thing I can say is that while subsets of a set are the
language of probability and of logic (the existential quantifier and the
universal quantifier are the two adjoints of the inverse image function), we
have the dual to be the language of entropy: partitions of a set.

You can unify the two via a self-dual set theory, where you essentially
replace the notions of subset (i.e., subobjects) and partitions (i.e.,
quotient objects) with an abstract subobject notion. This was part of the
basis for my MSc and then my supervisor and a colleague of mine [1] showed
that you can have what they call a Noetherian form over the category of sets.
It is essentially by taking the pullback of the subobject and quotient object
functors, though I can't remember if it is the usual pullback when seen as a
category of categories.

[1] This is not all published, but you could have a look at the papers of
Zurab Janelidze. The key point was his work on functorially self-dual group
theory.

------
tuxxy
It's funny seeing this at the top of HN today. I just wrote a utility in Rust
this past weekend to calculate the Shannon and metric entropy of a file
([https://github.com/tuxxy/entropy/](https://github.com/tuxxy/entropy/)).

I really love the Shannon entropy equation. I don't know why, but I just find
it so fascinating and elegant. I love it so much that I even got it tattooed
on my wrist recently haha!
([https://twitter.com/__tux/status/1257737842308022275/photo/1](https://twitter.com/__tux/status/1257737842308022275/photo/1))

------
birktj
I like the original paper by Shannon "A Mathematical Theory of Communication"
([http://people.math.harvard.edu/~ctm/home/text/others/shannon...](http://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf))
a lot. It is quite readable and also probably the most important paper in the
field of information theory.

------
jzer0cool
I was introduced first to "gray code" as an undergraduate for error
correction. For those new to this topic it could be where you can start for an
introduction.

