
Nobody Understands Probability - kmod
http://jsteinhardt.wordpress.com/2010/09/13/nobody-understands-probability/
======
frossie
For those who didn't make it all the way down:

 _People often intuitively think of probabilities as a fact about the world,
when in reality probabilities are a fact about our model of the world._

~~~
tome
There are at least two interpretations of probability. Firstly the
epistemological, as you say, represents our lack of knowledge about the world.
Secondly the aleatory truly represents the phenomenon of chance in the world.

The best interpretation of quantum theory for example (as I understand it)
takes the latter view that randomness is genuinely physically manifested, and
does not simply represent our inability to model reality.

~~~
scott_s
I think you're confusing concepts. The original point is about the difference
between the _map_ and the _territory_. And, because I'm (finally)
systematically going through Eliezer's sequences:
<http://wiki.lesswrong.com/wiki/Map_and_Territory_(sequence)>

Your second paragraph is about a particular map: quantum theory. Quantum
theory has probabilities in it. The dominant interpretation of quantum theory
is that the probabilities accurately represent what happens in the universe;
they are not artifacts for us to correct. But there is still a difference
between our map (quantum theory) and the territory (the universe itself).

Put another way: quantum theory is a map with uncertainty baked into it. But
this uncertainty has been _accurately mapped_.

~~~
tome
No I don't think so. Unlike statistical physics, where probabilities are
simply a mathematical technique for dealing with uncertainty, quantum
mechanics actually postulates that randomness is inherent to the universe.

If you disagree with me, please describe how your concept of "maps and
territories" applies to the StatPhys/QM distinction.

~~~
scott_s
Suppose I accurately map the coastline, and every relevant part of the
coastline is depicted in my map. But the map is _not_ the same as the
coastline itself.

If my coastline has some feature that blips in and out of existence in a
predictable way, I can integrate that into my map. My map then has uncertainty
in it. That uncertainty is an accurate reflection of the coastline itself -
but there is still a distinction between the map and the coastline.

I don't disagree with your second sentence. But there is still a difference
between our theory of quantum mechanics and the universe itself.

------
scottw
The odds of _nobody_ understanding probability is near-zero...

~~~
JacobAldridge
I figured there was a good chance someone would say that.

------
adolph
This essay by Yudkowsky is also helpful.

<http://yudkowsky.net/rational/bayes>

------
RickHull
> _However, the answer is not, in fact, 1/3. Why is this?_

This seems like a canard to me.

Here is my defense of 1/3 as a correct answer: <http://gist.github.com/578386>

> _Is Bayes’ theorem wrong?_

> _No, the answer comes from an unfortunate namespace collision in the word
> “given”. The man “gave” us the information that he has at least one male
> child. By this we mean that he asserted the statement “I have at least one
> male child.” Now our issue is when we confuse this with being “given” that
> the man has at least one male child, in the sense that we should restrict to
> the set of universes in which the man has at least one male child. This is a
> very different statement than the previous one. For instance, it rules out
> universes where the man has two girls, but is lying to us._

No, we are assuming that the givens are facts that are true.

> _Even if we decide to ignore the possibility that the man is lying, we
> should note that most universes where the man has at least one son don’t
> even involve him informing us of this fact, and so it may be the case that
> proportionally more universes where the man has two boys involve him telling
> us “I have at least one male child”, relative to the proportion of such
> universes where the man has one boy and one girl. In this case the
> probability that he has two boys would end up being greater than 1/3._

No, we don't have to consider universes where the man has at least one male
child but does not inform of us of this fact. We have a set of givens that are
assumed to be true, and based on those givens and the rules of logic, we can
make justifiable statements of probabilities.

~~~
thwarted
1/3 isn't correct, and I find the OP's explanation to be overly complex.

The set of possibilities for two genders of two children is GG, GB, and BB. In
your possibilities, GB and BG are exactly the same set (order doesn't matter
in a set, only membership), so you don't have 4 possibilities, you have 3
total. Since the guy asserted that one of them is a boy, you can rule out the
GG possibility. This leaves only GB and BB as possible results, both of which
have a 1/2 chance of being the correct one. The guy never makes a claim that
the first child or the second child is the boy (but, this doesn't change the
possibility that he has two boys, it just changes which one you remove from
the possibilities based on the provided information).

I'm not sure that it's that people don't understand statistics (although I'm
not in a position to confirm or deny that), it's that people don't understand
set theory. At least if you're going to use this "genders of two children" as
example.

~~~
RickHull
> _The set of possibilities for two genders of two children is GG, GB, and
> BB._

Yes, but there are two equally probable paths to arrive at _(set-theoretic)
GB_. Each of these paths is equally probably to the remaining paths, _BB_ and
_GG_. There are 4 possible paths, and _(set-theoretic) GB_ is the result of 2
of them.

Your application of set theory is inappropriate given 2 independent events.

~~~
thwarted
The question isn't what is the probability of any one of the paths, the
question is about the probability of the final result.

A fork in the road that joins up again gives each fork equal probability of
reaching the destination.

~~~
seabee
You can't understand the final result in isolation. Out of all men with two
children, the probability that they will have two daughters is 1/4, the
probability that they will have two sons is 1/4. This leaves heterogeneous
offspring occurring 1/2 the time.

If the man is able to make the statement "I have two kids, at least one's a
boy", this puts him among the 3/4 of all men with one or two sons. The
probability of a man with two sons cannot jump from 1/4 to 3/8 (half of 3/4),
as you assert earlier.

It's unintuitive, but it's more obvious when you negate the statement: "I have
two children, but I do not have two daughters."

~~~
carbocation
Perhaps this would offer an equally appealing explanation: if the man said, "I
have two children but at least _my firstborn_ is not a girl..." then the
intuitively appealing response of 1/2 becomes roughly correct.

------
jmtulloss
If you're lucky enough to attend UIUC, I highly recommend taking ECE 413 to
get a thorough introduction to these concepts. It's unfortunate that none of
the class materials are online since it goes well above and beyond what is
taught in most undergraduate CS courses on statistics. Taking it was hard, but
it made me a much better engineer.

Edit: I suppose this applies to anybody in college. Take the hard statistics
course that goes over this stuff. It's really valuable, and pretty hard to
pick up on your own.

------
rmathew
A more thorough introduction to this topic is "Probability Theory: The Logic
of Science" by E. T. Jaynes (<http://www-biba.inrialpes.fr/Jaynes/prob.html>).

------
equark
This article is not particularly clear. It doesn't have a clear discussion of
Bayesian versus frequentist interpretations of probability or inferential
statements that are conditioned on the unobserved true parameter versus the
observed data. It's hard to understand the subtlety of probability without
understanding p(theta), p(x), p(theta|x) and p(x|theta).

~~~
aphyr
I've read several pieces on Bayesian stats, and I've done some nontrivial
statistics before. It _still_ confuses me that p(data) != 1. I kinda wish the
author had gone into detail about how to calculate the probability of an
already-observed event.

~~~
tel
You're confusing p(data) with p(data|data) which is, trivially, equal to 1.

p(data) is better formulated as p(data|F) where F codifies your assumptions
about the possible generative probability models that you're building your
likelihood function from. Or, similarly, F codifies your understanding of the
world and the possible things that could occur within it.

This makes p(data|F) a perfect normalizing constant for the numerator of
Bayes' Theorem since the numerator implies a choice of a _specific_ model in
the family F, but p(data|F) averages over all possible models/worlds/parameter
choices (contained in F).

------
iliketosleep
'Let’s consider an example. Suppose that a man comes up to you and says "I
have two children. At least one of them is a boy." What is the probability
that they are both boys?'.

Am I missing something or in his attempt to solve the problem, does he
implicitly assume statistical dependence?

If statistical independence is assumed, with P(Boy) = P(Girl) = 1/2, then the
answer to the problem is very simple. P(Boy | Boy) = P(Boy) = 1/2.

Maybe I just don't understand probability :(

~~~
tel
The basic formation (which the author argues is not subtle enough to be true)
is better thought of step by step.

 _Suppose a man comes up to you and says "I have two children"_

At this point you build a set of possible realities, your model. There are
four possibilities: {BB, BG, GB, GG}. This space fully describes a model
whereupon there are two distinct, children with genders. Additionally, via
assumption of independence and equal likelihood, you can assign probabilities
to each observation, {BB:1/4, BG:1/4, GB:1/4, GG:1/4}.

 _"At least one of them is a boy."_

At this point, you update your realities by removing the one firmly
contradicted by the new evidence. Your new space is {BB, BG, GB} and when you
renormalize the probabilities you get {BB:1/3, BG:1/3, GB:1/3} which leads to
the idea that the probability at this point that the man has two boys is
1/3rd.

The author suggests however that during that second step, you should also take
into account the possibility that this guy is lying or that the fact that he's
proffering this information actually changes the likelihoods of those four
scenarios in a way different from just multiplying one of them by 0. So
perhaps the likelihood of hearing "At least one of them is a boy" is reflected
like this:

{BB:0.35, BG:0.32, GB:0.32, GG:0.01}

And your new belief in each of these realities reflects that like so
(renormalized)

{BB: 0.35, BG:0.32, GB:0.32, 0.01}

So now I feel even more confident that he has two boys.

~~~
iliketosleep
Thanks for your explanation. I think his first renormalization process is
wrong. Because once we know there is a boy, the problem space is reduced do
"What's the probability of a boy?" which is 1/2. It has nothing to do with
probabilities involving the known child.

~~~
aphyr
The problem space is not reduced to the gender of the unspecified child. The
important distinction is that "One of my children is a boy" is a statement
about _both_ children, not just one of them. Compare that statement to "My
first-born child is a boy," and it may make more sense.

~~~
iliketosleep
I appreciate your reply. But I'm still not clear on it. Because of this: "At
least one of them is a boy." As I see it, this statement contains the
following pieces of information: 1\. There are two children. 2\. One of them
is a boy. The question is... what's the probability of there being two boys?
Considering the information we've got, there are two possible scenarios
remaining 1. [B, B] 2. [B, G]. So we have P = 0.5. No?

~~~
ced
Three scenarios remain: [B,B], [B,G], [G,B], so the answer is 1/3.

Maybe you were thinking that the order doesn't matter. In that case, what was
the probability of getting a boy and a girl, in any order? 1/4 + 1/4 = 1/2. So
that's still twice as likely as getting two boys, and that ratio (2:1) will
still hold after eliminating [G,G]. You again get 1/3.

Some people find it easier to picture it in terms of frequencies. Imagine 1000
families. What fraction of them have two boys, among those that have at least
one boy?

~~~
iliketosleep
ced, yes i was thinking that the order doesn't matter. I think this is what it
comes down to. Do you think that order matters? If so, why?

yes i do find it easier to picture in terms of frequencies. in this case, take
1000 families which fulfill the criteria of "2 children with at least 1 boy".
what is the probability that a family will have 2 boys? we have not sampled
randomly. we have sampled according to the "2 children with at least 1 boy"
criteria. we are not dealing with two random variables. one variable is fixed
and we sampled according to it. now we are working with one independent random
variable within that sample. that random variable has P = 1/2.

is there a flaw in my logic? if there is, please highlight it. i think the
main confusion is: 1\. we have sampled according to particular criteria. 2\.
we need to calculate a probability within that sample. NOT the population that
sample was taken from.

~~~
tel
In a sampling of 1000 families, the expected values of each kind of family is
as follows:

    
    
      2xB : 250
      1xB, 1xG: 500
      2xG : 250
    

Sampling this population ignoring any family that has no boys leads to the
probabilities

    
    
      2xB : 1/3
      1xB, 1xG: 2/3rds
    

You're still looking at the same probabilities; the models agree.

I don't fully understand your two random variables formulation. I think the
confusion you're getting at is that there is an assumption that the chance of
any given birth being male is theta = 0.5. The question however is not

"I have two children, at least one is a boy, what is the probability that my
next child is a boy?"

It instead has to do with binomial probabilities on the space of a few
repeated trials under parameter theta. The distribution is no longer flat.

Here's a more stark example of a similar form.

"I have 300 children, and at least 1 is a boy. What are the odds that I have
no girls?"

~~~
iliketosleep
ok, you stated this very well. It's now clear where the confusions arises:
"Sampling this population ignoring any family that has no boys." Yes, with
this interpretation the answer is 1/3, but it's contrary to my interpretation.

Actually, for anyone who is interested, see "Boy or Girl paradox". There is
literature on this which discusses the different interpretations.

~~~
tel
That sampling arises because being part of the population which has no boys is
* necessary and sufficient* to (truthfully) make the statement that forms the
paradox.

The alternative interpretation of the paradox arises when the wording of the
paradox is construed to _identify_ one of the children as male or female. In
this case (stating something like "my first child is male"), being part of the
population (x \in {BB, BG}) is necessary and sufficient and leads to the 1/2
probability of having two boys.

In short, the question becomes whether you believe the child is identified in
the wording of the question. Honestly, the author of the paradox goes pretty
far out of their way to say "at least one of the children is male" avoiding
that identification.

------
signa11
since we are talking about probability-theory, thought folks here might find
gnedenko pretty interesting: [ [http://www-history.mcs.st-
andrews.ac.uk/Biographies/Gnedenko...](http://www-history.mcs.st-
andrews.ac.uk/Biographies/Gnedenko.html) ]

------
AmberShah
"Nobody"? That's not likely...

------
drakep
Never use absolutes?

