
The quiz Daniel Kahneman wants you to fail - bootload
http://www.vanityfair.com/business/features/2011/12/kahneman-quiz-201112.print
======
Uhhrrr
The engineer/lawyer problem explanation

(spoilers)

is only correct in a world where lawyers and engineers have all the same
characteristics. That the sampled individual is a man will skew things all by
itself.

Of the lawyers, approximately 40% will be women, whereas only 11% of the
engineers. So our samplee could be one of 27 engineers or 42 lawyers - we've
already bumped Peng from .3 to .39! That he likes math puzzles easily takes
Peng to over .4, meaning the answer can't be A).

~~~
bfung
30 engineers, 70 lawyers, the probability of being an engineer is 30%. What
Jack likes to do is irrelevant. Why can't a lawyer like math and dislike
politics (winning a case framed by certain rules is just a puzzle/game to
hack)?

~~~
prof_hobart
It's not that a lawyer can't like maths or dislike politics. The issue is
whether they are less likely to on average. From my personal experience of
knowing several of both groups, I would say that on average the lawyers I know
are less interested in maths than the engineers and more interested in
politics than them. It doesn't apply universally (some of the engineers I know
are obsessed by politics, just not all of them and not as many of them as
lawyers). It's possible that there may be research that demonstrates this
doesn't hold when you look at the sum total of lawyers and engineers, but
that's not what the author is trying to rely on.

Assuming that there _is_ a skew of preferences, then this info isn't
irrelevant you can perfectly reasonably use this to help identify the
likelihood of this person being in one group or another. It doesn't guarantee
that you're right, but it will improve your chances.

~~~
condiment
I think the point that many are missing is that is not known for certain that
an engineer is more likely to enjoy certain hobbies over others. People use
their personal experience to develop a heuristic which this test is designed
to reveal.

Getting hung up over the specificity of the hobbies and interests and the
likelihood of those hobbies and interests representing either or a lawyer or
an engineer is irrelevant, because the only factual data that was provided by
the questioner is that 30% of the participants were engineers, and 70% were
lawyers.

~~~
prof_hobart
It's not "getting hung up" about the specifics. They are relevant, and clearly
deliberately chosen. You're right that we don't know this likelihood with
absolute 100% certainty, but that doesn't mean we should dismiss our personal
experience, and a bit of logic (maths is typically more useful for, and a more
practical path into, engineering than law, and there's far more politicians in
my country with a background in law than there are with a background in
engineering) out of hand.

What this article is trying to present is heuristic errors - like question 1,
where ignoring of the fact that sample size is relevant gets you to the wrong
answer. Ignoring the likelihood that there is a correlation between personal
interests and career choice seems to me to be the equivalent heuristic error
for this question.

Let me give you an alternate example. There are roughly 700 million Europeans
and roughly 300 million Americans. If I randomly picked one person out from
this, gave you no other information and asked you where they came from, you'd
have a 70% chance of guessing correctly by saying "Europe". If I told you that
their first language was English, that they loved American football and
baseball and hated soccer, and that their favourite TV show was Conan, and
then asked you to guess where they came from, it would be hugely naive to
ignore that information and still assume that they were probably European.
Yes, it's entirely possible that there are Europeans who fall into all of
those things, and I've not done a survey to find the exact percentage of each
group that answer this description, but I'd be prepared to put a fair amount
of money on the fact that there's a larger overall number of Americans who
answer it than Europeans, so the smart guess would now be that they are
American.

------
colanderman
The questions involving "90% chance of $1000 or 100% chance of $900" always
bother me. I never understand why economists think that a rational actor would
consider them equivalent; they're _not_ , unless you are making that choice
many many times.

But if I'm given that chance _once_ (which is presumably what most
participants assume, since that's not a choice that comes up often in one's
life), it's really then a choice between "90% of a ton of free money or 100%
chance of a ton of free money". Unless the dollar amounts are radically
different, who in their right mind would take the choice that could possibly
leave them _without_ a life-changing sum the next day?

~~~
2arrs2ells
The interesting thing is not that people are risk averse (and thus choose 100%
chance of $900), the interesting thing is that people become risk seeking when
it comes to losses (and thus choose 90% chance of -$1000).

You're creating a bit of a straw man when it comes to behavioral economists
and their view of "rational actors" - the whole field is built around the
understanding that there is more to economic decisions than expected value.

~~~
X-Istence
Yes, however there is a slight chance of winning the odds and not paying
anything. The risk is minimal. If the numbers were further apart I don't think
the same logic would kick in.

A = $900 lost B = $9000 lost with a 90% chance

Those are odds I wouldn't want to try. I would take the instant loss knowing
that my odds are just not good enough to win if I were to choose B.

~~~
neonkiwi
The reason that $1000 at 90% odds and $900 at 100% odds are used in this
example is the expected value is the same in both cases, making the situations
'equivalent'.

A 90% chance of losing $9000 has an expected value of -$8100.

~~~
erichocean
The way I read it is this:

Do you want to be guaranteed you'll lose $900?

Or do you want a 10% chance you'll lose nothing at all, with a 90% chance
you'll lose another $100?

So given a choice between being (nearly) totally wiped out, or having the
chance of not being wiped out, people take the chance of keeping their cash.

Makes sense to me.

~~~
mseebach
It seems odd to me that this disproves Bernoulli theory "that a person’s
willingness to gamble a certain amount of money was a product of how that
amount related to his overall wealth".

If I could pull $900-$1000 from my savings with no immediate consequences, I'd
be more likely to spend the $900 at 100%. But if loosing $900-$1000 means I'll
have to tell my landlord I'll be late with the rent and then finding someone
to borrow it from, and paying it back with interest, the extra $100 aren't
significantly more crippling - it's the transaction cost of going through all
this bother that's problematic - I'll take a 10% chance.

Come to think of it, I actually did something like this: Prior to moving
abroad a while ago, I consulted a lawyer to make sure I did everything right
to avoid double taxation. That was a taking on a 100% chance of a rather big
expense to avoid an unknown chance of an even larger expense.

------
jimrandomh
Kahneman's work is great, and deserves to have attention directed to it, but
this Vanity Fair article is pretty bad. Why does the title say that Kahneman
wants people to fail the quiz? That's wrong, and since it's attention-grabbing
in a way that the truth isn't, I'd say it's dishonest. And the rest of it is
riddled with errors and confusions.

So go read Daniel Kahneman's books, and don't read Vanity Fair.

------
jonathlee
Despite the article's imputation that heuristics is a quick vs. accurate trade
off, Gerd Gigerenzer's work show that in fact it is usually "quick AND more
accurate" when used in the real, "large", world versus the "small" world of
games and logical puzzles where all of the rules are known and knowledge of
the problem is perfect, with infinite time allowed for optimization and
calculation. The video here is well worth watching to counter-balance
Kahneman's focus on edge cases where heuristic's break down or the wrong
heuristic was applied.

<http://videolectures.net/ijcai2011_gigerenzer_heuristicus/>

~~~
aangjie
+1 for pointing out the contradictory research.....

------
usaar333
As someone who has had lots of economics/probability training, this quiz
really doesn't do the research justice. It gets a point across, but could have
been presented better. I was bothered by its misuse of terminology and the
quiz not really being one.

Spoilers ahead!

1\. The answer could be A or C, depending on how small and large the hospitals
are. The small hospital could easily be expected to have < 5% more variance
(depending on definition of "small" and what 5% of each other means) depending
on its and the large one's size.

2\. I don't see why the correct response must be 30%. Such attributes
conditionally describe an engineer better than a lawyer. Assuming random
sampling of lawyers and engineers, I'd be surprised if the answer isn't > 30%.
I'm not sure what the numbers are actually, as I don't know how strong this
conditional applies to engineers or lawyers.

The point of the research is that people over-emphasize the conditional over
the prior (indeed often ignore the prior), not that people should not use
conditional information.

3\. " it is likely that your answer to question (a) is positively correlated
to your answer to question (b)"

I understand what they are trying to say, but the wording is quite off.
Correlation is a property of data, not an individual point. If I'm the only
person who ever takes this test, my answers have undefined correlation. The
modifier "likely" is especially baffling (correlation is a constant property
of data).

Proper (and less verbose!) terminology is "people's answers to (a) are
positively correlated to (b)"

------
tikhonj
Not exactly the same, but my favorite visceral examples of imperfect
heuristics are "garden-path sentences"[1]. These are sentences that trick you
into having to backtrack when you're parsing them, which brings the whole
parsing process into sharp relief.

[1]: <http://en.wikipedia.org/wiki/Garden_path_sentence>

A good example is:

    
    
        The old man the boat.
    

Figure out why this sentence is actually grammatically valid and then read the
article.

~~~
SilasX
To save everyone the trouble of reading a lot to learn a little: "man" is a
verb in that sentence.

Though if I wanted to be clever I could hammer that into less garden-pathy
sentence as follows:

"In my will, I left everything to the one who could use it best. I gave the
boy the legos. I gave the girl the dollhouse. I gave the mother the kitchen
set. The father the hunting rifle. The bachelor the suit. The old man the
boat."

------
rayiner
A good summary paper on behavioral economics by a professor at Caltech:
<http://www.hss.caltech.edu/~camerer/ribe239.pdf>

Dan Kahneman, Richard Thaler and Dan Ariely have both published some popular
books on the subject. Thinking Fast, Thinking Slow just came out last year.
Predictably Irrational came out a few years ago. Nudge (Cass Sunstein and
Richard Thaler) looks at the implications of behavioral economics for the law.
There is a good summary article of that work here:
[http://www.law.harvard.edu/programs/olin_center/papers/pdf/2...](http://www.law.harvard.edu/programs/olin_center/papers/pdf/236.pdf).

I personally think this work is pretty earth-shattering in the field, and that
the above work is a must-read for anyone interested in economics. The engineer
side of me is really attracted to the fact that behavioral economics uses
legitimate experimental methodology, instead of mathematically-supported
handwaving. And the implications of the work really turn some of our
assumptions about the nature of the economic system on their head.

~~~
NeilCJames
Much earlier ('91, I think), but a better read for those with a technical
mindset (but still accessible) is Thaler's The Winner's Curse.

------
repsilat
I'm afraid I don't quite understand the point of the second question. Does the
answer mean to state authoritatively that that engineers are no more likely to
do carpentry and partake in recreational mathematics than lawyers, or is there
another explanation?

If it were changed to something like

"2. A team of psychologists performed personality tests on 100 professionals,
of which 50 were engineers and 50 were nurses. Brief descriptions were written
for each subject. The following is a sample of one of the resulting
descriptions:"

"'Jack is a 45-year old man.'"

and if we were to assume that 80% of engineers are men, and that 80% of nurses
are women, we'd expect 80% of the men to be engineers (and so expect Jack to
be an engineer with 80% probability). Maybe?

Question 4 - the theatre ticket - could be worded more clearly. I thought "as
you enter the theatre" meant "as you enter the big room with the stage and
lots of chairs". Who would buy a ticket after entering that room, no matter
what the circumstance?

~~~
SatvikBeri
The question is really wrong. I've heard a similar question before and it goes
more like this:

"A team of psychologists performed personality tests on 100 professionals, of
which 5 were engineers and 95 were lawyers. Brief descriptions were written
for each subject. The following is a randomly picked sample of one of the
resulting descriptions: 'Jack is a 45-year old man who enjoys recreational
Mathematics.' Assume 90% of engineers enjoy recreational Mathematics and only
10% of lawyers do. What is the chance that Jack is an engineer?"

And the point is that the odds are still that Jack is a lawyer. The question
as posed in the article, though, really doesn't make sense.

------
atuladhar
I recently read Kahneman's 2011 book "Thinking, Fast and Slow" -- it should
probably be required reading for everybody who's in charge of making decisions
that affect a lot of people.

~~~
jacques_chester
I have to second this advice, even though I'm halfway through it.

Kahneman has spent decades investigating biases and cognitive error. And it
turns out he's not all that bad at popularising his and related research.

Of particular fascination (and frustration!) are the little examples he
liberally sprinkles throughout the book. Small quizzes, questions and the like
for the reader to try. Try as I might, I have consistently "failed" these in
the way Kahneman goes on to explain.

He even explains why he does that: because it's hard to particularise from the
general. It's easy to generalise from the particular.

I've found that many dimly held intuitions I've developed over the years about
"how the world works" are starkly illustrated in the book. And many things I
believed were not borne out be research.

Bottom line: it's a great book and very illuminating. Read it.

~~~
nkurz
I'm halfway through it as well, but have a less positive impression. I came
into it with a lot of respect for Kahneman and his research, but the first
half of book is extremely flat. Much like the reaction here to this Vanity
Fair teaser, I constantly find myself quibbling with the examples, and
disagreeing with the explanations.

I'm not sure who to blame, though. I think the book was written over a
considerable period of time. Perhaps Kahneman's standards have changed?
Perhaps there were multiple editors involved, some of whom (like the Vanity
Fair author) didn't really understand the material? Or perhaps the editor was
great, but only worked on some of the chapters.

I'd offer a much less enthusiastic bottom line: It's a frustrating book, but
you should read it anyway. Skip to Chapter 20 "The Illusion of Validity" if
you get bogged down.

~~~
jacques_chester
I agree that the writing style is flat, but it's also very thorough and
methodical. More than once I've found myself thinking "but what about X?" and
then there, on the very next page, he addresses X to a depth I'd never
considered.

The biggest counterargument against his tests is that like a lot of sentences,
they can be read in more than one way. Possibly for people with System 2 minds
that like to dissect concepts, that's enormously annoying.

------
mixmax
His nobel prize lecture is well worth a watch if you have 37 minutes to spare.

<http://www.nobelprize.org/mediaplayer/index.php?id=531>

~~~
sesqu
This lecture revealed an error in question 5.

The Vanity Fair article says "Kahneman and Tversky debunked Bernoulli’s
utility theory, a cornerstone of economic thought since the 18th century.
(Bernoulli first proponed that a person’s willingness to gamble a certain
amount of money was a product of how that amount related to his overall
wealth—that is, $1 million means more to a millionaire than it does to a
billionaire.)"

But in the lecture, Kahneman stated no qualms with utility theory. Rather, he
pointed to the application of the theory, where Bernoulli had assumed a gain
of $1 million is equivalent to $1 million appearing in their bank account,
with the mechanism being irrelevant. That is what prospect theory (and the
associated question) debunks.

------
Vivtek
I didn't much like this: " _being swayed by the way in which questions are
worded rather than responding just to their substance_ " (for the lost ticket
being allegedly _equivalent_ to $10).

Is a ticket, which costs $10, emotionally equivalent to $10? Once bought, the
ticket is unique in my eyes, whereas I'm not even sure how many $10 bills I
have in my wallet even now. So if I lose one, well, maybe it wasn't there in
the first place.

I guess what I'm trying to say is that the impact of the ticket loss and the
impact of loss of a $10 bill are _not_ equivalent in terms of substance.
Unless you're an economist.

~~~
yariang
I would dare say unless you're rational, not necessarily an economist. Most of
us are not rational all the time, and the whole point of the question is to
show an instance where we are not.

You would be correct if there were not other tickets available; in that case,
the ticket truly is unique. But in the example given you could acquire another
one. In that case the ticket is not unique and is equivalent to the cost of
acquiring another one, which is $10.

~~~
chrischen
But I believe it is the duty of the ticket vendor to keep track of who bought
tickets. In protest, I would not replace a ticket if I lost it, but if I lost
$10 it is completely my responsibility. Is this not rational if I believe I
can affect change through the action?

~~~
yariang
That's an entirely different issue than the one I was responding to. In that
case, it would be rational if your actions could reasonably affect change,
which in this case I don't think they would.

First, I don't see an obvious need for theaters to keep track of who purchased
tickets--save the case of online transactions obviously. Granted, the whole
example is contrived--but I don't see the utmost need for it.

And, even if they had a moral imperative to do so and didn't, this would
mostly affect people--by definition--who purchased a ticket already, and
either lost it or want to return it or something of that nature. It is safe to
assume few people lose their tickets and of those that do, not all of them
protest by not buying another one. Protests are usually only effective if they
hurt the company in terms of reputation or money. Since that is unlikely
(given the vague probabilities I mentioned and specially because you already
paid for one ticket), your protest will most likely be in vain.

We could of course get into a philosophical debate over the worth of such
protest, but this is neither the time or place. Protesting the theater record-
keeping policies in the way you mention and for the reasons you mention would
probably be, in my opinion and with all the information you provided,
rationally irrational.

~~~
chrischen
Um... if I buy another ticket then I help them. So not buying them denies the
sale. So it DOES have a measurable effect.

------
sdizdar
The paper they wrote in 1983 is a must read for all entrepreneurs (and sales
people in general): [http://pol-
sci.uga.edu/courses/2008/ainsworthfall08kahnemant...](http://pol-
sci.uga.edu/courses/2008/ainsworthfall08kahnemantversky.pdf)

------
kstenerud
Their interpretation of question 5 is just plain wrong. Aside from the fact
that losses DO hurt more than gains (because you need to alter your plans to
accomodate losses), the assumption of equivalence is fundamentally flawed.

The problem is that the two halves are inverted. Correcting:

5a: Your base position is $900, and you have the option of risking $900 for a
possible gain of $100. You have a 90% chance of gaining $100 (a trivial
amount) and a 10% chance of losing $900 (a big amount).

5b: Your base position is -$900, and you have the option of risking $100 for a
possible gain of $900. You have a 90% chance of losing $100 (a trivial amount)
and a 10% chance of gaining $900 (a big amount).

Since you're an individual, and not a statistical average, there's no middle
ground in the values gained or lost. So the rational person is right to choose
A, then B. The two are NOT equivalent to an individual. The trivial amount
makes little difference as a gain or a loss, but the big amount does.

------
jpiasetz
I don't understand the title? Why does it matter either way if someone fails
or gets it right?

~~~
yariang
If most people answered all or most of the questions correctly, it would be
disproving Kahneman and Tverksy's theory of how people think.

The title assumes they would hate to be proven wrong, but makes sense.

------
joshu
I also recommend reading Fortune's Formula.

I took a stab at Prospect Theory myself:
<http://joshua.schachter.org/2008/09/amateur-economist.html>

Took a few hours of work.

------
berdario
What does "How many dates did you have last month?" even means?

Maybe I've a different notion of "date" from the rest of the people... but I'd
assume that if you've a stable relationship with a significant other, then
your number could be definitely higher than 5 in a month...

otherwise, if it's considering all the different dates with all potential
partners, 3 to 5 seems an number exceptionally high (in fact: I've got a grand
total of 0 dates over the last 24 months, and I'm quite sad about that)

------
kia
Question three about dates is pretty much Anchoring Effect:

<http://en.wikipedia.org/wiki/Anchoring>

Also an interesting reference about anchoring and electric cars from the above
Wikipedia article:

[http://www.treehugger.com/cars/nissans-leaf-creates-a-new-
an...](http://www.treehugger.com/cars/nissans-leaf-creates-a-new-anchor-for-
electric-car-pricing-forcing-others-to-match-it.html)

~~~
icefox
The thing is question three is about dates and question three _B_ is about how
happy you are so the two are suppose to be related. If it was question three
and question four maybe, but not when they put the two questions together.

------
taylorbuley
My "choice & behavior" focus in college was pretty obscure at the time so it's
cool to see Prospect Theory and other concepts start to go mainstream.

------
janj
I've started reading 'Thinking, Fast and Slow' and really like it so far. If
anyone in the Bay area wants it after I'm done let me know.

------
DannyPage
"The framing effect is also used to explain the influence of positive and
negative information on our decisions—for example, why consumers prefer to buy
ground beef labeled 80 percent lean rather than 20 percent fat."

Then why don't politicians say "Employment rates are at 91%" instead of the
depressing "Unemployment rates are at 9%"?

------
paganel
Sorry to say it, but this certainly smells like cargo-cult science. First, I'm
not a scientist, and not even a Nobel-prize winner, but for example I cannot
understand what question 3) has to do with "science". I did have 2 dates in
the past month, but I'm at one of the lowest points of my life, because, guess
what? the missing context really matters (I'm about to get divorced).

The same goes for question 4) . I chose A, "Yes", assuming it was a good movie
worth seeing, but lacking the above-mentioned "context" (is it "Forrest Gump"
or "Delta Force IV"?) any answer you give it's just to satisfy the
interviewer. And yes, people DO go and see movies that they know they're bad.

A little less reading on "behavioral economics" or "animal spirits" and a
little more reading of Hume will do everyone a ton of good.

~~~
nl
The questions aren't science - that statistics that predict that most people
will get them wrong are.

Or rather - the theories which predict how people will (statistically) make
sub-optimal decisions in certain circumstances _are_ science: they make
predictions that can be tested.

~~~
Radim
I believe paganel is referring exactly to the "sub-optimal decision" part, in
the explanation (not the questions themselves).

And yes, this quiz seems a load of underspecified cargo-cult horsecrap. I
surely hope the original research was of higher quality.

~~~
nl
_I believe paganel is referring exactly to the "sub-optimal decision" part, in
the explanation (not the questions themselves)._

Can you expand on that? The point of the article is that people will
(statistically) get the questions wrong, or at least have biases in their
responses.

For example, in the first question the grandparent poster may well have had
good reasons to answer as they did, but that doesn't discount the fact that
most people are vulnerable to the attribute substitution bias[1].

 _And yes, this quiz seems a load of underspecified cargo-cult horsecrap._

That's to be expected isn't it? They are trying to show five different
statistical effects in five questions. To do it properly they would have to
ask 10's of questions for each one, but I don't think that would work for a
Vanity Fair sidebar article. Instead, they tried to find questions that would
demonstrate the principles to most people.

[1] <http://en.wikipedia.org/wiki/Attribute_substitution>

~~~
Radim
Sure.

What I meant (and what I believe paganel meant) is _not_ that most people
won't get such questions wrong, statistically. They probably will, and the
original research probably shows that conclusively.

It's the part of this quiz/post that is supposed to explain _why_ that irks
scientifically inclined folks (as manifested many times in this thread :) Here
the author missed crucial bits that made the question+explanation originally
work, and all that's left is some well-meaning cargo cult nonsense.

Whether it appeared in a sidebar or not is no excuse for getting it wrong imo
-- they could have stuck with the original questions if they didn't understand
the implications of modifying their premises.

------
gizzlon
The ticket questions is dumb. Has anyone heard a better version of it?

At least to me, the important factors are: Do I remember where I'm suppose to
sit? Would It be immoral to attend even if I lost the physical stub?

------
srl
After the first question: after being primed not to trust my instincts, I
decided (SPOILER) it couldn't be (a), and thus guessed (c). Which I suppose
reflects another important cognitive bias.

------
nl
And here's the reason why people who say economics isn't science are (mostly)
wrong: economics is (mostly) applied statistics.

Yes, interpretation of the statistics isn't "just science", but when is it
ever?

~~~
burgerbrain
Excuse my ignorance, but what does the scientific method have to do with
statistics?

I have always considered it to be a math, not science.

~~~
nl
I regard Math as a Science, although I do understand the controversy.
[http://en.wikipedia.org/wiki/Mathematics#Mathematics_as_scie...](http://en.wikipedia.org/wiki/Mathematics#Mathematics_as_science)
is a pretty good summary.

(Although if economics is Math then that is even better: economics is
something that can make predictions, and _isn't_ just the black magic that
some seem to think)

The scientific method applies is deciding what statistical experiments to do.

