
The Man Who Invented Information Theory (2017) - huihuiilly
http://bostonreview.net/science-nature/tom-rutledge-man-who-invented-information-theory
======
Cybiote
Claude Shannon's Masters Thesis was on the application of Boolean Algebra to
circuits, effectively founding digital circuit design. That should have been
enough for anyone, but not for Shannon. His later work on Information theory
has proven important in everything from evolution to quantum mechanics
(particularly, relative quantum entropy) and perhaps even to future physics
([https://www.youtube.com/watch?v=td1fz5NLjQs](https://www.youtube.com/watch?v=td1fz5NLjQs)).

He also did early work in cryptanalysis, AI (minimax chess algorithm and a
learning robotic mouse). His student was Ivan Sutherland. More or less,
whether its networking, signal processing, compression, crypto, machine
learning, circuit design, basically anything to do with the digital age,
you'll find Shannon did important foundational work there.

~~~
porpoisely
It is a shame shannon doesn't get the recognition he deserves. Computer
science has many fathers and as amazing as Turing was, nobody contributed more
and nobody has a better claim to call himself the father of computer science
than Shannon. Not Turing. Not Church. He outstrips them all but nobody outside
of the computer science world has heard of him.

~~~
Wistar
Just two nights ago, on Amazon Prime streaming, I watched the very good show
"BBC Order And Disorder Episode 2 - Information" hosted by the excellent Jim
Al-Khalili. He profiles Alan Turing and says that Turing was only half the
story and then introduces Claude Shannon—a name with which I was
unfamiliar—and Bell Labs. Great stuff.

~~~
atombender
Great documentary. I can heartily recommend all of Al-Khalili's documentaries:
"Atom", "The End of the Universe", "Shock and Awe: The Story of Electricity"
and "Chemistry: A Volatile History" are all excellent. The last two are my
favorites; while his documentaries on physics and quantum mechanics are good,
they describe scientific phenomena that are pretty difficult to understand,
let alone visualize, and it ends up having to be a little over-simplified, or
explained with unsatisfying visual analogies. The ones on chemistry and
electricity are a lot more (no pun intended) grounded, and include some fun
demonstrations.

~~~
Wistar
Wow, thank you. I will seek out those shows.

~~~
mooreds
I did some googling and it looks like you can see at least some of this stuff
online: [http://www.documentarytube.com/videos/chemistry-a-
volatile-h...](http://www.documentarytube.com/videos/chemistry-a-volatile-
history)

~~~
Wistar
Thank you. I discovered last night that some of the shows you mention are not
on amazon Prime. Only the more astronomical and physics-related ones are
(Gravity; Order/Disorder; Everything/Nothing; Beginning/End of the Universe).

------
lixtra
"A Mathematical Theory of Communication" is a beautiful read. You easily find
the PDF but once the latex source was available as well - in case you want to
reformat for e-book reader:
[https://web.archive.org/web/20130129025547/http://cm.bell-
la...](https://web.archive.org/web/20130129025547/http://cm.bell-
labs.com/cm/ms/what/shannonday/paper.html)

~~~
crdrost
I cannot overstate this point and upvote it enough.

The reason Claude Shannon is a _legend_ has a lot to do with the fact that his
ideas are not just _correct_ and draw from his multidisciplinary knowledge,
but also _expertly communicated_. It is an incredibly readable paper and only
absolutely requires a little bit of mathematics when he is describing how to
convert a state machine for Morse code to a matrix which would allow someone
to calculate how many bits per time unit can actually be transmitted in Morse
code -- and this is not absolutely essential, it is calculated in a different
way and also it is communicated that you can easily take his word for it that
it is whatever numerical value it was. A lot of the arguments about fitting a
stream of symbols into an encoding for noisy channels have these lovely
diagrams that help to elucidate exactly what he's talking about.

If you want to make that same sort of impact, it is not just important to be a
great mind, but to spend a bunch of time practicing how you communicate that
information.

~~~
x3tm
Exactly. Effective communication is what made Feynman a legend.

Sadly, many extremely smart and profound scientists are quite incapable of (or
not interested in) conveying clearly their thoughts to general audiences.

~~~
mehrdadn
I guess it's fitting that a man who created a theory of communicating
information was effective at communicating information.

~~~
r00fus
It’s actually not that common and a common fallacy. We even have a phrase “the
tailors kids have the worst shoes” indicating that experts and geniuses don’t
necessarily apply their own logic because application is different from
theory.

------
Zanneth
The ingenuity and brilliance coming from Bell Labs during that era is
absolutely astonishing. Transistors, information theory, satellite
communication, UNIX/C, the list goes on. These ideas unquestionably laid the
foundation for modern high-tech society.

If anyone is interested in learning more about Bell Labs and the folks who
worked there, “The Idea Factory” by Jon Gertner is a fantastic book written on
the subject. It’s not comprehensive but it’s a very inspiring read.

There has always been something very vexing about Bell Labs’ legacy though.
They had everything they needed to start the personal computing revolution:
engineers, scientists, equipment, a nationwide telephone network for god’s
sake. What happened?

~~~
cfallin
I heartily second the recommendation for "The Idea Factory"! I'm currently
reading this book and aside from the seriously impressive run of successes,
the characters are really quite amusing at times too. E.g., Shannon built a
desktop calculator that operated using Roman numerals only ("THROBAC") in
order to amuse himself. And some of the whimsical creations were pretty
impressive in their own right. E.g., his maze-solving mouse "Theseus" learned
the maze layout on progressive runs through the maze by using relay-based
logic.

~~~
mindcrime
I'll happily "third" the recommendation for _The Idea Factory_. It really is
an inspiring read, and I expect many firms could find some actionable
takeaways to apply within their own organizations, even today. Yes, Bell Labs
had some specific circumstances that allowed them some luxuries that other
firms may not have, but their success is not as simple as "they had a
monopoly, herp derp".

------
adolph
_A new Shannon biography, A Mind at Play: How Claude Shannon Invented the
Information Age, may help reverse this legacy. Authors Jimmy Soni and Rob
Goodman make a strong bid to expose Shannon’s work to a popular audience,
balancing a chronological narrative, the “Eureka!” moments that sprang from
his disciplined approach to solving puzzles, and his propensity for
playfulness._

Another book with a lengthy section on Claude Shannon is James Gleick's The
Information: A History, a Theory, a Flood [1.]. A shorter but still nice
explanation in on Brain Pickings [2.].

1\. [https://www.goodreads.com/book/show/8701960-the-
information](https://www.goodreads.com/book/show/8701960-the-information) 2\.
[https://www.brainpickings.org/2016/09/06/james-gleick-the-
in...](https://www.brainpickings.org/2016/09/06/james-gleick-the-information-
claude-shannon/)

~~~
hinkley
Is he really unknown? I know most of us don’t do much signal processing these
days but you can’t get far in studying compression without hearing about
Shannon.

~~~
QML
Shannon may be well known in the HN community, but definitely not outside of
here.

~~~
usrusr
Surprising plot twist: I first heard of Claude Shannon from a music magazine
which I read instead of doing my CS homework.

------
Isamu
Shannon was a speaker at my commencement at CMU. He struck me as kind of a
self-effacing, amusing sort of character. Maybe like a much quieter, less ego-
driven version of Feynman.

Also: as much as people talk about information theory in various contexts, I
doubt that many take the trouble to understand it better. Like thermodynamics,
people want to take away an overly broad, folksy interpretation and apply it
everywhere without stopping to think if it really applies in the way they
claim.

~~~
shaklee3
I'm not sure what you mean here, so it may be worth elaborating. I think a lot
of people understand how theory, and certainly in communication theory
everyone understands the famous equation. I don't see anyone claiming it
applies in places it doesn't.

~~~
Isamu
Shannon himself talked about it in "The Bandwagon":
[https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1056774](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1056774)

>I don't see anyone claiming it applies in places it doesn't.

It applies in a lot of places, but probably in a more narrow way than people
think. You can say "information theory applies here" and probably be right,
but to be specific about what that means requires some work. Shannon says that
himself in that piece.

Thermodynamics is an easier one to see, because people always make claims
about something "because of the 2nd law of thermodynamics", something about
everything always becoming more disordered. But that's too simplistic a view,
and fails to consider what is a closed system and so on.

------
garf_
Another great book (about Shannon and others) is "The Idea Factory: Bell Labs
and the Great Age of American Innovation"

[https://books.google.com/books/about/The_Idea_Factory.html?i...](https://books.google.com/books/about/The_Idea_Factory.html?id=uOMt_XCo81QC)

------
JoeAltmaier
Ok his fundamental theorems aren't strictly binary. He wrote of discerning
possible 'symbols' from noisy measurement where a symbol could have N possible
values. The optimum discernment was at 2 values (1 and 0) but his math handled
any base.

If I remember right after 20 years.

~~~
hervature
Actually, if I recall correctly after only 5 years, your first sentence is
correct. It doesn't matter how many symbols your alphabet has (i.e. base
system used in application) because they can all be represented using binary.
I don't recall anything about "optimum discernment" just that it is convenient
to standardize proofs using a binary alphabet. For example, Huffman coding is
proved using binary but it easily applies to ternary systems as well.

~~~
JoeAltmaier
Yeah it had to do with Signal-to-Noise Ratio (SNR) being best at 2 symbols I
think.

~~~
DoctorOetker
>The optimum discernment was at 2 values (1 and 0)

I think this "optimum discernment" at base 2 you remember is nonsense? could
you provide a pointer?

------
an-allen
"Anything can be communicated over a noisy channel without error" \- Was
something my Elec 201 professor would confidently proclaim before he would go
into a lengthy aside about how amazing Claude Shannon is.

That was nearly 20 years ago.

Everyday I appreciate that statement, and Claude Shannon, a little bit more.

------
crb002
He also showed the limits of the efficient market hypothesis in economics. Add
in Einstein (information cannot travel faster than the speed of light), and
Kolmogorov (random information cannot be compressed) to get 90% of why markets
cannot be fully efficient.

------
argonium
There's a fantastic book on Shannon called "Grammatical Man", by Jeremy
Campbell -
[https://www.amazon.com/dp/B000KHFL0W](https://www.amazon.com/dp/B000KHFL0W)

~~~
ThomDenholm
While this book was originally published in 1973, it contains a lot of great
information that is relevant today. I should re-read it...

~~~
dredmorbius
FYI, 1982, not '73.

[https://www.worldcat.org/title/grammatical-man-
information-e...](https://www.worldcat.org/title/grammatical-man-information-
entropy-language-and-
life/oclc/856698430/editions?start_edition=31&sd=desc&referer=br&se=yr&editionsView=true&fq=)

------
insaneirish
It's been forever since I read it, but Fortune's Formula [1] is an
entertaining read that goes into Shannon's investment methodology, alongside
Edward Thorp [2].

[1]: [https://www.amazon.com/Fortunes-Formula-Scientific-
Betting-C...](https://www.amazon.com/Fortunes-Formula-Scientific-Betting-
Casinos/dp/0809045990/ref=sr_1_1?ie=UTF8&qid=1548815625&sr=8-1&keywords=fortune%27s+formula)
[2]:
[https://en.wikipedia.org/wiki/Edward_O._Thorp](https://en.wikipedia.org/wiki/Edward_O._Thorp)

------
jshowa3
One of my heroes. I have his picture as my phone background so that I'm
constantly reminded of the minds that built the device I use everyday.

When I saw that masters thesis. I literally hyperventilated.

------
nicholast
There's a kind of paradox of shannon entropy found in communications with
double-entendre I think, in that they appear to increase both redundancy and
entropy. I touched on that in this essay proposing what i call subtractive
adversarial networks.

[https://medium.com/@_NicT_/subtractive-adversarial-
networks-...](https://medium.com/@_NicT_/subtractive-adversarial-
networks-7465c3e161e5)

------
pstew
The biography they reference, "A Mind at Play: How Claude Shannon Invented the
Information Age", is on sale in the Kindle store right now:
[https://www.amazon.com/Mind-Play-Shannon-Invented-
Informatio...](https://www.amazon.com/Mind-Play-Shannon-Invented-Information-
ebook/dp/B01M5IJN1P/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=&sr=)

~~~
tzs
Not just at the Kindle store. The eBook is also on sale at Apple Books, Barnes
& Noble, BAM!, Google Play, and Kobo. They are all $3.99, same as the Kindle
sale price.

------
JoeDaDude
Claude Shannon had an interest in AI and had a hobby of making electronic game
machines, most of which survive to this day in the MIT Museum. A list with
descriptions and photos is available here:

[https://www.boardgamegeek.com/geeklist/143233/claude-
shannon...](https://www.boardgamegeek.com/geeklist/143233/claude-shannon-man-
games-and-machines)

------
yters
The odd thing about Shannon information is a coin flip can have just as much
information as text, yet the former would intuitively seem devoid of
information and the latter information rich. It is unclear what exactly is the
relationship between Shannon information and what we intuitively consider
information. Any ideas?

~~~
princeofwands
Shannon information deliberately only concerns syntactic information
(information content). Other, more recent work, focuses on semantic
information (information with meaning for a receiver).

> Shannon information theory provides various measures of so-called "syntactic
> information", which reflect the amount of statistical correlation between
> systems. In contrast, the concept of "semantic information" refers to those
> correlations which carry significance or "meaning" for a given system.
> Semantic information plays an important role in many fields, including
> biology, cognitive science, and philosophy, and there has been a long-
> standing interest in formulating a broadly applicable and formal theory of
> semantic information. In this paper we introduce such a theory. We define
> semantic information as the syntactic information that a physical system has
> about its environment which is causally necessary for the system to maintain
> its own existence. "Causal necessity" is defined in terms of counter-factual
> interventions which scramble correlations between the system and its
> environment, while "maintaining existence" is defined in terms of the
> system's ability to keep itself in a low entropy state.

[https://arxiv.org/abs/1806.08053](https://arxiv.org/abs/1806.08053)

Roughly speaking: The amount of computation or energy needed to perfectly
reproduce a random source, such as a coin flip, is high, while the
significance or meaning, for the average receiver, is low. Natural language
text requires less computation to reproduce [1], but, for the average
receiver, the significance is higher.

[1]
[http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf](http://languagelog.ldc.upenn.edu/myl/Shannon1950.pdf)

~~~
yters
Hmm, but you could compress text to an equally random sequence. I.e. all
minimal programs are by definition Kolmogorov random.

Also, what about crystalline forms, which are very orderly and require minimal
computation to reproduce, but are equally insignificant for the average
receiver?

~~~
princeofwands
> you could compress text to an equally random sequence

More or less correct. The key difference is that you could not compress a
random coin flip sequence (and that a compressed text is meaningless until
decompressed to original).

> all minimal programs are by definition Kolmogorov random

Compression provides an upper bound to K. Kolmogorov Randomness itself is not
computable. AKA: You can't ever know if you have a minimal program.

> Crystalline forms

It is possible to both have low significance and low information content.
Crystalline forms were very significant to Turing though:
[https://en.wikipedia.org/wiki/The_Chemical_Basis_of_Morphoge...](https://en.wikipedia.org/wiki/The_Chemical_Basis_of_Morphogenesis)

~~~
yters
The best approach that I've seen is a combination of Shannon information and
Kolmogorov complexity. If an object has high Shannon information, then it is
not crystalline. If it also has low Kolmogorov complexity then it is not
random. This seems to characterize the sweet spot where meaningful information
occurs. Kolmogorov called this quantity "randomness deficiency".

~~~
princeofwands
Really cool concept. Thanks!

------
achillesheels
This is a controversial opinion but: Claude Shannon is more noteworthy than
Einstein. It’s simply the case engineering science is not taken seriously by
the scientific community despite its overwhelming contributions to human
development, namely digital signal processing and computer engineering.

------
victornomad
Information Theory was one of my favorite subjects @ uni. Wish I had more time
to dig in now!

~~~
llamaz
What did you cover?

At my uni they only had communications theory, which covered stuff like
software defined radios. Information theory was a significant part of it
though.

------
pbhowmic
I would argue Shannon’s case is far from unique. For my money, Gauss was the
greatest mathematical mind of all time but does the general public know of
him.

------
vixen99
Dover's excellent 'An Introduction to Information Theory' by John Pierce is
dedicated to Claude and Betty Shannon.

------
bitxbit
I read The Mathematical Theory of Communication every few years and find
something new every time.

