
The Man Who Invented Modern Probability - denzil_correa
http://nautil.us/issue/4/the-unlikely/the-man-who-invented-modern-probability
======
tikhonj
> _Kolmogorov armed a group of researchers with electromechanical calculators
> and charged them with the task of calculating the rhythmical structures of
> Russian poetry_

And this is what "big data" and NLP were like before computers :).

One can only imagine what he could have done with a modern computer. (Which is
particular hard to imagine because he's responsible for advances which
ultimately led to these same modern computers...)

Also, as an aside, I'm very impressed by the quality of nautil.us articles.
I've come a across a few randomly, and they've been uniformly high in quality,
interesting and insightful.

Moreover, unlike much popular science/math reporting, the articles are not
ridden with obvious errors and yet are easily followed by somebody not in the
field. An impressive combination. (Not surprising for this particular article,
I suppose: it was written by a professor teaching the history of mathematics
at MIT.)

~~~
dxbydt
>And this is what "big data" and NLP were like before computers :)

aside, I recently interviewed a candidate for a "Big Data" position. He got
quite chatty about Kolmogorov & Measure Theory, so I quickly cooked up an
interesting problem to relax his nerves. I opened the interview with "If the
side of a cube comes from a uniform distribution between 1 and 5 inches,
what's the expected volume of the cube ?" I thought he'll give me the answer
in 10 seconds & we'll move on but instead he was completely stumped & just
stared blankly. I asked him to use the whiteboard but he simply drew a cube
and wrote a^3, a~[1,5], and then went back to staring at it. Finally after 10
minutes of pin-drop silence, my partner, a non-math guy, took over the
interview & asked him some Hadoop related technology questions & he started
speaking again.

~~~
relaia
27?

~~~
lsconjugate
39

~~~
solve

        from random import random
        sum([(random()*4+1)**3 for x in xrange(1000000)])/1000000
        => ~39
    

Weird.

~~~
bluetech

      sum(random.uniform(1, 5)**3 for i in range(10000)) / 10000
    

Many people don't know that the random module allows to sample from many
distributions directly without having to construct your own.

~~~
hollerith
I like the first version better because everyone knows what infix * and + do.

~~~
saucetenuto
I like the second version better because it taught me something I didn't know.

------
sampo
The treatment of the Paradox of the Great Circle in the article is not very
good. If you're left wondering, Wikipedia helps:

[http://en.wikipedia.org/wiki/Borel%E2%80%93Kolmogorov_parado...](http://en.wikipedia.org/wiki/Borel%E2%80%93Kolmogorov_paradox#A_great_circle_puzzle)

~~~
lmkg
tl;dr Because Great Circles have measure 0, any treatment of probability on
them must instead use limits that asymptotically approach Great Circles. Also
because they have measure 0, such limits don't converge in a consistent
fashion. As it turns out, the two most common limit constructions give
different results for that particular problem.

The closest analogy I can think of is trying to assign a value to the
expression 0/0\. It doesn't have a value, but you can construct limits that
appear to converge to it. One such limit is lim{x/x} as x approaches 0, which
tends to 1. Another limit is lim{x log(x)} as x approaches 0, which tends
towards 0. Both of these constructions have claim to being the value of 0/0,
and they don't reconcile with each other.

------
consz
My father was the grand-grad student of Kolmogorov. He (Kolmogorov) is a very
impressive mathematician. My father's advisor, Yakov Sinai, is equally
impressive and arguably one of the founders of Dynamical Systems, a very
interesting subfield of mathematics (and, in some ways, probability theory) in
its own right.

~~~
dev1n
Do you mean Kolmogorov-Sinai entropy? I just read about this in _Chaos Theory
Tamed_ by Garnett P. Williams. Really interesting stuff with respect to
information theory in dynamical systems.

------
te_platt
An interesting take the Kolmogorov system can be found here:

[http://www-biba.inrialpes.fr/Jaynes/cappal.pdf](http://www-
biba.inrialpes.fr/Jaynes/cappal.pdf)

In it E. T. Jaynes compares Bayesian statistics Kolmogorov probabilities and
finds them essentially identical in result even after working from very
different first principles.

~~~
wamatt
For anyone interested, the above appendix comes from ET Jaynes's _[1]_ magnum
opus, _" Probability Theory: The Logic of Science" [2]_.

Which, IMHO is deserving of it's grand title and provides an excellent
framework for using Bayes in the real world.

_[1][http://en.wikipedia.org/wiki/Edwin_Thompson_Jaynes](http://en.wikipedia.org/wiki/Edwin_Thompson_Jaynes)
_

_[2][http://shawnslayton.com/open/Probability%20book/book.pdf](http://shawnslayton.com/open/Probability%20book/book.pdf)
_

------
richardjordan
I have no reason to particularly disbelieve the research that went into this
article. But I'm just naturally skeptical when he reels off a list of puns
that depend on the english language used by a group of Russian researchers.
This seems unlikely to me. It seems like unnecessary color of dubious
provenance.

------
pessimizer
Why haven't I heard of this journal? Looks great.

~~~
japaget
It's a startup! They started publishing about a month ago.

------
nawitus
>For example, irrational numbers—those that cannot be written as fractions—
almost surely have no pattern in the numbers that appear after the decimal
point. Therefore, most irrational numbers are complex objects, because they
can be reproduced only by writing out the actual sequence.

That's false. Pi is irrational, but there's many short algorithms to calculate
it's digits.

~~~
mag487
The key phrases there are "almost surely" and "most irrational numbers." There
are irrational numbers like pi which can be defined algorithmically, but taken
together as a set they have zero Lebesgue measure. (In fact, they're
countable.)

~~~
nawitus
I didn't know that. Any link to a proof?

~~~
nwhitehead
Consider the set P of programs that take no input and generate an infinite
stream of digits. Each program in P has a finite length and is written out of
a finite set of symbols, so there must only be a finite number of programs of
any given length. That makes P countable. Let Q be the set of numbers
described by programs in P. Each program from P describes exactly one number,
so Q must also be countable. The set of irrational numbers is uncountable, so
there must be an uncountable set of "complex" irrational numbers remaining
after you take away all the "non complex" ones in Q.

~~~
nawitus
Your proof seems to only prove that a finite number of programs (described by
you) which can produce a finite number of irrational numbers, while there are
infinite number of irrational numbers.

But we're surely not talking about a finite number of programs.

I'm not even sure what the Kolmogorov complexity of a "complex irrational
number" means. If you need the sequence of digits and you cannot use an
algorithm to produce the digits, then the complexity is infinite?

~~~
nwhitehead
For each fixed length there are a finite number of programs of that length. If
we use 8-bit bytes for the alphabet, there are 256^N programs of length N. The
set of ALL programs is infinite (we don't limit the length of programs), but
it is countable.

The "countable" part means we can put the set into one-to-one correspondence
with the natural numbers. The correspondence starts with 0 mapping to the
empty program, then 1-256 mapping to programs of a single byte, then 257-65793
mapping to the programs of two bytes, and so on. This mapping will hit each
program exactly once, and it will hit every program because every individual
program has a finite length.

Different types of infinities are not intuitive so don't feel bad if the
concepts are confusing. These issues troubled lots of very smart
mathematicians for decades. The existence of irrational numbers was hugely
troubling to Pythagoreans. The existence of uncountable infinities discovered
by Cantor was shocking [1].

[1]:
[http://en.wikipedia.org/wiki/Controversy_over_Cantor%27s_the...](http://en.wikipedia.org/wiki/Controversy_over_Cantor%27s_theory)

~~~
mtdewcmu
In some sense, the existence of irrational numbers with no pattern in their
digits is an illusory artifact of the number system. We can talk about them
collectively because decimals are not required to have an end, and we have to
postulate them to fill in the gaps in the number line, but such numbers lack
any description or means of being separated as individuals.

But then, is any mathematical abstraction real? I guess it's all beside the
point.

~~~
nwhitehead
> But then, is any mathematical abstraction real? I guess it's all beside the
> point.

I actually think the reality of mathematical abstractions is hugely important
because of... computer programs! In a real way, programs are the embodiment of
mathematics. I want my programs to work so I need the underlying math to work
as well.

That's why I'm a constructivist. I reject the law of the excluded middle
because proofs that use it don't translate into real programs; they translate
into programs that ask an omnipotent oracle to decide which branch to take.
Constructive proofs translate into working programs.

It also ties into philosophy. I am a skeptic, so when someone tells me either
A or not A must be true even if we can never know which one, I ask for proof
or justification of that fact. The justifications that I get are remarkably
similar to logical "proofs" that god exists, and just as fallacious. This
isn't to say that there couldn't be an ultimate truth about A or a god, just
that it is not logically necessary.

~~~
mtdewcmu
Interesting. I'm not all that up on philosophy, and I had to look up
constructivism and the law of the excluded middle. Computers can only deal
directly with discrete math, so that eliminates quite a bit of mathematics.
Moreover, despite being regarded as Turing machines, you could in reality
count all the states that a computer could be in, by counting the bits of
memory, and you will find that it's a finite number. So computers are, in
fact, only finite state machines. It's often overlooked that what makes a
Turing machine a Turing machine is the infinitely-long tape. So computers
occupy a fairly small sliver of the infinite universe of math.

Philosophically, my worldview is like that of science: the way to know
something is by making observations and formulating and testing hypotheses.
What we can observe is limited, and hypotheses are only models that are tested
by successive approximations. I'm not sure I understand what you mean that
that statement isn't required to have an ultimate truth. Presumably a
statement like that has an answer, but the fact that something is knowable in
principle doesn't mean that there's any way to get the answer. For instance,
the Hubble telescope can see far away galaxies that we can never get an up
close look at. The question of whether there's life somewhere else in the
universe must have an answer, but we can't know what's in those galaxies; even
with a better telescope, we'd be seeing what they looked like a billion years
ago. Many things will never be known.

------
buro9
I thought this was going to be about Pierre Simon Marquis de Laplace who wrote
A Philosophical Essay on Probabilities in 1814.

[http://neuroself.com/2012/01/21/pierre-simon-
laplace-1814-a-...](http://neuroself.com/2012/01/21/pierre-simon-
laplace-1814-a-philosophical-essay-on-probabilities/)

------
ceautery
I am not a statistician, but if I found myself in an infinite forest, I would
probably get drunk, too.

------
tlarkworthy
Discovered, not invented.

~~~
lutusp
> Discovered, not invented.

This refers to a philosophical debate that asks whether mathematics is part of
nature or an artificial invention of man. I think the debate has begun to lean
toward mathematics being part of nature, in which case yes, the proper term is
discovered.

~~~
bachback
Interestingly enough this notion, that there is only this one type of
probability was debated before Kolmogorov by people like Ramsey and Keynes.
Now the subjective interpretation of probability is called Bayesian
statistics, which is not a helpful term.

[http://plato.stanford.edu/entries/probability-
interpret/](http://plato.stanford.edu/entries/probability-interpret/)

------
kenko
How is this not about Francis Galton?!

~~~
anifow
For those not in the know, Galton invented the simplest way of understanding
normal distribution. Here's a copy you ca ceck on your phone.
www.intromath.ca/aakkozzll

A quote from Galton:
[http://www.aakkozzll.com/docs/order/galton.htm](http://www.aakkozzll.com/docs/order/galton.htm)

