
How Claude Shannon Re-Invented Information (2017) - dnetesn
http://nautil.us/issue/51/limits/how-information-got-re_invented
======
woodandsteel
This is the first thing I have read about information theory that really made
sense to me. I think I now understand it about ten times as well as I did
before.

>What does information really measure? It measures the uncertainty we overcome

My thought here is that matters because human beings have to make decisions
all the time, and reducing the uncertainty regarding the situation makes it
more likely you will make the correct decision.

~~~
lisper
There's a pithy slogan that sums it up nicely: information is a _correlation_
between the state of two systems.

For example: if you measure a voltage using a volt-meter then, if the volt-
meter is working properly, there will be a correlation between the readout on
the meter and the actual voltage, and so the meter contains information about
the voltage.

Note that information is symmetric: the voltage in this case also contains
information about the meter.

~~~
jonnydubowsky
Your comment and the one above it are awesome summaries of some of Shannon's
concepts. Re:your statement on information being symmetrical, how does zero
knowledge encryption, Homomorphic encryption and the like fit into this when
there is asymmetrical knowledge and validation without full access to the
information?

~~~
lisper
The best way to approach that question is to start with the one-time pad
(OTP). The OTP has a random key K which is the same size as the message M. The
ciphertext C is M XOR K.

So... if information resides in correlations, where does the information
reside in this system? Because K is random, it has zero correlation to M
(that's what "random" _means_ ). It also has zero correlation to C, though
this is not so obvious. (Proof is left as an exercise.) In fact it turns out
that C and K are both completely random, and so it's not even possible to tell
by looking at C and K which is which. They are interchangeable.

So we have the following situation:

1\. Information about M obviously has to reside _somewhere_ in the composite
system comprising C and K because we can completely recover M simply by
computing C XOR K.

2\. The information is not in C and it is not in K.

So where is it?

[Spoiler alert. It's worth thinking about this for a while to see if you can
figure it out on your own. Hint: think about the case where C, M and K are all
just a single bit, because the multi-bit case is just a straightforward
extension of the single-bit case.]

Answer: the information about M is contained in the correlations (or anti-
correlations) between C and K. In the one-bit case, if C and K correspond
(i.e. are both 0 or are both 1) then M is 1. If they don't correspond, then M
is 0.

The more complicated cryptographic cases are very similar, but of course, more
complicated. What cryptography is really about in some sense is trying to get
the same results but with a smaller key, i.e. it's about trying to take a
fixed-size K that is (much!) smaller than M and "distribute" the entropy in
such a way that the information is hidden in correspondences between bits in
such a way that an attacker can't figure out where those correspondences are.

~~~
jonnydubowsky
This is hands down the best Hacker news interaction I've had! I took a
screenshot of your explanation and did not scroll past the spoiler. Will give
this some thought before proceeding... Thanks!

~~~
lisper
BTW, I just realized that I got this part backwards:

> In the one-bit case, if C and K correspond (i.e. are both 0 or are both 1)
> then M is 1.

That should have read "... then M is 0." (Obviously.)

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=14978399](https://news.ycombinator.com/item?id=14978399)

~~~
a_imho
It is submitted by the very same user.

~~~
de_watcher
He keeps us informed.

~~~
oli5679
Maybe he's adding redundancy, because he's worried that Hacker news is a noisy
channel.

~~~
stronglikedan
_Or_ , maybe he's adding redundancy, because he's worried that Hacker news is
a noisy channel.

------
mkagenius
If anyone from Nautilus is here: Would you consider having different prices
for different countries? The current pricing is too high for India. A reduced
price would bring customers from various countries.

~~~
criddell
A lifetime subscription is $99. If you factor in life expectancy, they are
actually charging more to people in India than they are to Americans.

~~~
Anon84
Yes, if you look at raw dollars per year... but you also have to factor in
expected lifetime earnings. What fraction of your expected income is $99?

I would wager that it's a much higher fraction in India than in the US.

------
flopska
I recently read the Shannon biography 'A Mind at Play' [0] which I guess is
the full version of the article (same authors). It does (IMHO) a very good job
at explaining entropy / information.

[0] [https://www.simonandschuster.com/books/A-Mind-at-
Play/Jimmy-...](https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-
Soni/9781476766690)

~~~
goldfeld
I enjoyed the story but found the author(s) practically peddling Shannon's
importance as if any of that was needed. All in all the writing in form and
substance seemed to me of low quality and in search of dramatic revision and
editing. Great history job and lazy writing,

~~~
woodandsteel
>I enjoyed the story but found the author(s) practically peddling Shannon's
importance as if any of that was needed.

The vast majority of the public have never heard of Shannon, or have but don't
really understand how important he was, so yes, it was needed.

