

The King of Human Error - gruseom
http://www.vanityfair.com/business/features/2011/12/michael-lewis-201112.print

======
crntaylor
The quiz that accompanies this article [1] makes an error. Question 2 asks

"A team of psychologists performed personality tests on 100 professionals, of
which 30 were engineers and 70 were lawyers. Brief descriptions were written
for each subject. The following is a sample of one of the resulting
descriptions:

Jack is a 45-year-old man. He is married and has four children. He is
generally conservative, careful, and ambitious. He shows no interest in
political and social issues and spends most of his free time on his many
hobbies, which include home carpentry, sailing, and mathematics.

What is the probability that Jack is one of the 30 engineers?"

The answer given is that the probability is exactly 30%, since there are 30
engineers out of 100. But this ignores all of the information available in the
description! Imagine if the description said "Jack's hobbies include writing
compilers, hacking assembly code, World of Warcraft, designing new programming
languages and LARP." Would we still conclude that there's only a 30% chance of
him being an engineer?

A correct answer to this question should apply Bayes' rule. With E
representing 'Jack is an engineer', L representing 'Jack is a lawyer' and D
representing the description, Bayes rule tells us

P(E|D) = P(E) P(D|E) / P(D)

P(L|D) = P(L) P(D|L) / P(D)

where P(A|B) is 'the probability of A given B' and P(A) is 'the probability of
A without taking other information into account'. We can ignore the common
factor of 1/P(D) and we know that P(E)=0.3 and P(L)=0.7, so the relative
probabilities of Jack being an engineer or a lawyer, given the description of
him, are

P(E|D) ~ 0.3 * P(D|E)

P(L|D) ~ 0.7 * P(D|L)

so the a priori probabilities of Jack being an engineer or a lawyer need to be
weighted by the probability of seeing the description that we did, _given_
that Jack is an engineer or a lawyer! If you claim that there is a 70% chance
of seeing the description if Jack is an engineer but only 30% if Jack is a
lawyer, then taking all the evidence into account, you should ascribe an equal
probability to the two possibilities.

Basic probability fail by the editors of Vanity Fair, I think.

[1]
[http://www.vanityfair.com/business/features/2011/12/kahneman...](http://www.vanityfair.com/business/features/2011/12/kahneman-
quiz-201112)

~~~
ckuehne
Not only do they seem to lack basic probability skills. They also cannot cite
the original material correctly. Kahneman and Tversky tested something else
completely. Namely, that people tend to ignore base rates when making
probability estimates. From Kahneman's book:

"In one experimental condition, subjects were told that the group from which
the descriptions had been drawn consisted of 70 engineers and 30 lawyers. In
another condition subjects were told that the group consisted of 30 engineers
and 70 lawyerss. [..] In sharp violation of Bayes' rule, the subjects in the
two conditions produced essentially the same probability judgements."

There is also a blatant error in the OP article: Answer No. 2 (Linda is a bank
teller and is active in the feminist movement.) is by no means "logically
impossible".

EDIT: wording

~~~
three14
'There is also a blatant error in the OP article: Answer No. 2 (Linda is a
bank teller and is active in the feminist movement.) is by no means "logically
impossible".'

They meant that it's logically impossible for the probability of A AND B to be
higher than the probability of A, since A AND B is a subset of A. She could be
feminist, but the probability of being feminist is lower than the 100%
probability that she's either a feminist or not.

~~~
ckuehne
I am well aware of the point they were trying to make. Or, more precisely, the
point Kahneman et al. were trying to make. What I am concerned about is that
they, that is, the editors / writers of Vanity Fair, do not understand their
mistake. This is somewhat likely given the error they made described by the
grandparent comment.

Overall, I find it rather ironic to find such gross errors in an article that
sets out to educate the reader about "human error".

------
mturmon
A well-written (by Michael Lewis) article that should invite people interested
in economics and human or rational decision-making to get to know Kahneman's
work, which won the Nobel prize, better.

Engineers try to make rational decisions but probably we don't realize the
extent to which it's not really human nature to do so.

Teaser:

Amos [Tversky] and I once rigged a wheel of fortune. It was marked from 0 to
100, but we had it built so that it would stop only at 10 or 65 One of us
would stand in front of a small group, spin the wheel, and ask them to write
down the number on which the wheel stopped, which of course was either 10 or
65. We then asked them two questions:

Is the percentage of African nations among UN members larger or smaller than
the number you just wrote?

What is your best guess of the percentage of African nations in the UN?

The spin of a wheel of fortune had nothing to do with the question and should
have had no influence over the answer, but it did. “The average estimate of
those who saw 10 and 65 were 25% and 45 respectively.”

~~~
SkyMarshal
I wonder what the control group/s who was/were asked the questions without the
wheel came up with.

~~~
jonsen
Aren't the emerging two groups each others control group? What could you gain
from your proposed control experiment?

~~~
cdog46
That article, as written in Vanity Fair, is so poorly written. Paragraph one
is evidence enough. More indefinite modifiers so that you don't know who is
who. Also I don't understand why answer 1 is the correct answer regarding
Linda. There is simply not enough information given to determine which answer
is true.

~~~
andrenotgiant
Regarding the Linda the Bank Teller Question: Here is an analogous numbers
only probability question:

You are told that the test administrator is thinking of a number between 1 and
100, and that the number is odd. What is more likely?

A. The number is greater than 50. B. The number is greater than 50 and ends
with a 3, 5, 7 or 9

Answer A and you have a 50% chance of being correct. Answer B and you only
have a 40% chance of being correct.

------
joshu
a while ago I did a mechanical turk experiment that shows experimentally a bit
of how kahneman/tversky prospect theory ends up working:
<http://joshua.schachter.org/2008/09/amateur-economist.html>

------
kstenerud
"Which alternative is more probable?

(1) Linda is a bank teller.

(2) Linda is a bank teller and is active in the feminist movement.

The vast majority—roughly 85 percent—of the people they asked opted for No. 2,
even though No. 2 is logically impossible. (If No. 2 is true, so is No. 1.)
The human mind is so wedded to stereotypes and so distracted by vivid
descriptions that it will seize upon them, even when they defy logic, rather
than upon truly relevant facts."

Umm.. no. The reason 85% of people chose option #2 is because the answers were
so poorly worded. They give the impression of an unspoken assumption for
answer #1 that Linda is NOT active in the feminist movement.

~~~
vilhelm_s
Various alternative explanations like that have been explored, but it seems
the effect is not due to wording:
[http://lesswrong.com/lw/jj/conjunction_controversy_or_how_th...](http://lesswrong.com/lw/jj/conjunction_controversy_or_how_they_nail_it_down/)

~~~
kstenerud
Actually, I would take it to mean that the wording has a huge effect on how
people respond, as per his previous post:
<http://lesswrong.com/lw/ji/conjunction_fallacy/>

Notice that in the jazz playing accountant example, 92% of people answered A,
E, C.

Notice now that in the gambling situation, only 65% of subjects answered 2.

92% for the "story" question, and 65% for the "hard data" question. I still
look at the jazz question and read it through the cultural interpretation of A
> E > C, despite it being incorrect in a strictly logical sense. That does not
make my interpretation a conjunction fallacy, regardless of what the question
creator intended.

This only goes to show that the conjunction fallacy has no hope of being
measured accurately unless the question is completely unambiguous in subtext
(meaning that it doesn't tread upon cultural aspects of language, where people
fill in the blanks and read between the lines).

And even then, the dice rolling question, though devoid of cultural baggage,
still might not be the best way to measure the conjunction fallacy. I know a
great number of people who couldn't figure out probability to save themselves.
This could have skewed the results higher or even lower, depending on what
proportion of the test subjects could properly interpret probabilities. Now if
this question were asked to statisticians, the results would be compelling.

------
JoeAltmaier
"They had a rule of thumb, he explains: they would study no specific example
of human idiocy or irrationality unless they first detected it in themselves."

God I love these guys.

~~~
danielharan
Also love this passage:

“Most people after they win the Nobel Prize just want to go play golf,” said
Eldar Shafir, a professor of psychology at Princeton and a disciple of Amos
Tversky’s. “Danny’s busy trying to disprove his own theories that led to the
prize. It’s beautiful, really.”

It shows amazing intellectual honesty.

------
boas
> Which alternative is more probable? > (1) Linda is a bank teller. > (2)
> Linda is a bank teller and is active in the feminist movement.

If Linda is a randomly chosen person, then clearly 1 is more probable.

However, there is another sense in which 2 is more likely to be true. For
example, in the radiology department where I work, doctors ordering studies
need to enter in a reason for the study. Frequently, they put something
generic like "pain" to get the study approved, which may or may not be
accurate. On the other hand, if I saw a history that said "1 day history of
RLQ pain, nausea, and fever", I would consider that history to be more likely
to be true.

So in a narrow probabilistic sense, 1 is true, but when you consider that
people sometimes make stuff up or don't have the full story, you might
rationally judge a more detailed story to be more believable.

------
SteveSailer
The Vanity Fair staffer, Jaime Lalinde, who made up the sidebar quiz,
misinterpreted Kahneman's point through the lens of political correctness:
Stereotypes (even about engineers) Are Wrong!

------
dombercz
cliffs?

