
Adventures in Cognitive Biases - srijan4
http://web.mit.edu/~xiac/www/advbiases/
======
Mithaldu
I don't know where this is going, but so far it's cute.

There's one major gripe i have though, the UI doesn't let me accurately
express my beliefs.

On the question "How many dollars does the average American spend at vending
machines during a year?" i thought it might be somewhere around 100, between
10 and 1000. However i could only get it as far as 38 - 162, which didn't
allow me to cover the right answer. Optimally though i'd like to be able to
answer "i haven't got a friggin' clue".

On the other hand, i guess it demonstrates a massive cognitive bias on the
side of the makers of this: Not everyone is confident in knowing the answer to
something.

Difficulties on other questions:

    
    
      - cat sleep hours in a day: allowed me to set ranges like 0-48
      - degradation time of a plastic bag: did not allow me to set 200-5000

~~~
dthunt
Yeah, this is a fair criticism. There are explicit bounds encoded in each
problem and when those conflict with your estimates, you will be frustrated.

Haven't got a frigging clue isn't a fair thing to say though. The whole point
of the exercise is that most people have a significant misalignment going on
between their modelling of reality and the track record of their modelling
ability.

Exercises like figuring out how much milk a cow produces in a week or how fast
a horse runs require you to consider what information you DO have suggests
about upper and lower bounds and things like that, and then correctly
realizing the confidence of your models is the other half of the equation.

You are pretty sure, I would imagine, that a cow doesn't produce 10,000
gallons of milk in a week. You are pretty sure that a 'heavy' user of a
vending machine might use three a day and buy a bunch of snacks and a drink or
two, and that 'heavy' users maximally represent Y% of the population, with
high confidence.

You can use these intuitions to guide your estimates, and you do in fact do so
on a regular basis. The exercise is calibrating those estimates against their
empirical accuracy.

~~~
Mithaldu
The point here is: The website asked me about my beliefs. It did NOT ask me to
work out the correct solution. And if i believe i cannot possibly give a
useful estimate then that should be a viable answer i can give.

Your example is especially relevant since i don't even know what a gallon is.
I guess it's maybe a liter or something? Who knows. Either way: Without actual
research there is no more accurate modelling of my belief than "i don't know",
period.

------
scott_s
They asked how many "reincarnations" there have been of the Dalai Lama. I
think, and maintain, that the answer is 0, as I don't think "reincarnation" is
possible. Had they instead said, "How many people have there been that have
been called 'The Dalai Lama'", I would have answered differently.

Since this is a game about cognitive biases, hosted at mit.edu, I maintained
hope that they, too, would be this pedantic about semantics, but, sadly, no.

~~~
tzs
If they had asked for an estimate of how many people died when the first Death
Star was destroyed, would you answer be 0 because there was no actual Death
Star?

~~~
scott_s
No, because such questions, to me, implicitly acknowledge we are talking about
a fictional world. The question about the Dalai Lama, however, is a question
about _this_ world.

Another example that I would say 0 to is, how many times did Jesus Christ come
back to life? But I would say 1 to, according to the Bible, how many times did
Jesus Christ come back to life? The difference between Star Wars and the bible
is that there are not large groups of people who insist that Star Wars is
real, so I feel safe in assuming an implicit, "According to this fictional
thing". I consider the "reincarnations of the Dalai Lama" to fall under the
bible category.

~~~
aptwebapps
"No, because such questions, to me, implicitly acknowledge we are talking
about a fictional world. The question about the Dalai Lama, however, is a
question about this world."

Or, perhaps you're implicitly talking about the world in which the Dalai Lama
is reincarnated. There's not much difference to your distinction.

The correct answer is whatever is commonly excepted by Buddhists, I guess, or
perhaps there is an official number. 0 would be a trick answer.

------
rahimnathwani
The overconfidence/underconfidence issue is similar to that expressed in the
book 'How to measure anything'. The author claims to have trained people to
become better estimators, by presenting them estimation challenges, and
providing feedback about how often the actual value was within their stated
range. IIRC people were asked to present a range for which they were 80%
confident. If the answer was within the range too often (e.g. 90%) then their
ranges were too wide, and if not often enough, they were overconfident. Over
time, most subjects improved their calibration.

I started (yesterday) working on a simple web app to train people in this way.
It's not yet ready to try out, but you can bookmark it here:
[https://github.com/rahimnathwani/measure-
anything](https://github.com/rahimnathwani/measure-anything)

------
AdrianRossouw
I think it's cute, but the overconfidence bias bit in the beginning is
frustratingly geo-centric.

It's all general knowledge questions using imperial measurements, dollar
spending power and american supermarket stats.

This needs a _shrug_ button.

~~~
GhotiFish
Yah, I feel like I'm being accused of being overconfident when my answer is "I
don't know"

------
samatman
The math here is problematic. I happened to know how many 'reincarnations' of
the Dalai Lama there have been (hint: he's routinely referred to as the
[redacted] Dalai Lama, much like Louis the Sun King gets a number). This gave
me ten thousand points. Other guesses give me fractions of a point.

What am I supposed to learn from that?

edit: checking again, ten thousand, not a thousand. corrected.

------
jsweojtj
as an astrophysicist, this one annoyed me:
[https://www.dropbox.com/s/udoug25cpd2zpiy/Screenshot%202014-...](https://www.dropbox.com/s/udoug25cpd2zpiy/Screenshot%202014-07-13%2009.29.21.png)

~~~
gravity13
Haha. I did the same exact thing.

------
MaysonL
US cows produce twice as much milk as the average, I guess. And their
assumption that everything is to be normally distributed is weird, and a
distinct cognitive bias in and of itself.

~~~
nazgulnarsil
yes, at the very least you should first move the circle vertically to define
the bounds, then move the circle horizontally to create the shape of the
distribution.

------
bluecalm
It has potential but they need to fix two things:

-make standard units available: liters, kilograms, meters;

-allow users to express full range of my beliefs; I have no idea how many reincarnations of Dalai Lama there was I guessed 1000 and wanted to set a straight line from 1 to infinity (or something going down to 0 slowly). It didn't allow me to; it's not overconfidence from my side, it's a bug on your side :)

~~~
judk
Do you really have no idea?

Do you believe in a likely lower bound for the time gap between
reincarnations? And an upper bound for when the first one was? Or do you
believe that both of these could be arbitrarily small (large, resp.)?

------
nmac
I think this is a very interesting game with an important message. However,
the UX could be more engaging by having a more vertical leveling up structure.
I felt as if I wanted to get to the next level but the onslaught of questions
seemingly never ended- after a bit this becomes disengaging, specially when
you aren't shown how many more units are needed to graduate.

~~~
gravity13
In case you're curious where it ends, you first go through the belief charts.
Then you start working with a little probability calculator. It ends with an
embedded google form with open-answer fields for responding to harder text
questions.

------
ISeemToBeAVerb
I'm not sure who this game is aimed at, but if you're trying to teach
cognitive biases to people with no training in psychology you really need to
add a tutorial to the beginning. I would have loved to go further with the
game, but I became frustrated at the very first question when I was confronted
with this:

"Drag down the circle so that your belief covers all answers that you think
are reasonable. This diagram is your belief graph."

I have no idea what a "belief graph" is, or how to use one. I though I had
answered the question by entering a number into the field, but then I'm
presented with this strange graph.

Perhaps I'm in the minority around here when it comes to my knowledge of this
domain, but if this is a game intended for laypeople, you need to be really
clear and assume I know nothing at all about your methods and practices.
Explain to me what a belief graph is and how to use it. Either have a pre-game
tutorial, or use tool tips.

That's my 2 cents.

~~~
gravity13
It becomes clear once you progress through a few of the examples what this
means.

You don't need tutorials and tool-tips, you just need to be able to learn by
doing.*

*and users who don't shy away on the first hint of confusion.

~~~
dthunt
Yes.

The problems absolutely illustrate the concept if you have a reliable
overconfidence confidence bias. You do not need a background in psychology to
understand "Hey, I feel 90% confidence, but am being surprised nearly 1 in 3
times. There's some kind of miscalibration going on there."

------
aristus
The green cab question was changed from the original in a way that (I think)
changes the logic.

[http://heatherlench.com/wp-content/uploads/2008/07/bar-
hille...](http://heatherlench.com/wp-content/uploads/2008/07/bar-hillel.pdf)

In the original, witnesses are only 80% accurate reporting color. In this
version it's altered so that it's simply asserted that 80% of cabs in the
neighborhood of the accident are green.

With that revision it seems to me that you would not update your priors but
actually replace them. "The mix of cabs is 85/15\. Oh wait, it's actually
20/80 the other way."

Edit: Ah, never mind. The paper gives both versions, specifically because the
in second version you are supposed to throw away the prior.

------
kriro
It was fun but the end was too much of a cliffhanger for my taste.

There should be a choice of metric/imperial and some of the questions were a
little meh. I liked the Bayesian part.

------
gravity13
I'd be interested to see the results of the first part of this. I've heard of
those studies (perhaps they were just stories?) where groups of people
collectively estimated things like number of jelly beans in a jar and were
actually pretty accurate on average.

Obviously it's not going to be like that, because some of these things people
can actually know - but I'm still curious.

~~~
dthunt
If you can find that, provide a link. I strongly doubt people are particularly
good (without applying some sort of rigor) at estimating beans in jars.

~~~
geekingfrog
This effect comes from the effect of regression to the mean. Some people will
vastly overestimate, while some people will vatsly underestimate, thus
cancelling out.

It holds as long as the people used are very diverse (not under the same
cognitive bias)
[http://www.socialresearchmethods.net/kb/regrmean.php](http://www.socialresearchmethods.net/kb/regrmean.php)

~~~
dthunt
If you ask people to estimate the number of protons inside the sun, you might
develop a peak around some sorts of numbers, but it's not likely to be a
particularly good estimate.

~~~
gravity13
Well, yeah, we all have a cognitive bias in that we've never had to comprehend
a scale at 10^35-10^45 before (that's my belief graph btw).

------
wodenokoto
I thought the test at the end was way too difficult given the simple
introduction to calculating probabilities.

~~~
MacsHeadroom
Yes, I came here to say the same thing.

~~~
dthunt
I am going to challenge that belief by asking you to do the following:

Try solving the blue and red pots problem, given only a single observation.

Now try again, this time with two observations.

------
spokenn
What do you think about Daniel Kahneman's 'Thinking Fast and Slow'?

------
judk
Arbitrary weird GUI control that doesn't work on mobile and doesn't have an
accessible fallback mode, even though the information content can be easily
expressed with short simple text: yep.

------
PeterGriffin2
I think the interaction is very interesting and engaging, lots of potential
there.

But the only thing I learned is to look up answers to trivia questions on
Google, and to drag chart bars to match the text above.

Overconfidence isn't the problem people have when obviously faced with
questions they don't know the answer to (I've never milked a cow, and I don't
intend to), and the inferences the protagonist was making from his trip were
very suspicious and arbitrary, so that was disengaging in a tale about using
proper logic.

~~~
sitkack
Dude, you cheated.

~~~
dthunt
Not only cheated, but also destroyed an opportunity to learn something
valuable.

GP can, however, take other calibration tests elsewhere.

Here is one:
[http://calibratedprobabilityassessment.org/](http://calibratedprobabilityassessment.org/)

Don't cheat this time. There is a valuable lesson here.

~~~
sitkack
I wimped out, the list was too long and it is really hot.
[http://imgur.com/OB890bo](http://imgur.com/OB890bo) at least my line is
mostly straight, ;).

Lots of those questions were ambiguous within a couple hundred miles, at least
in my mind.

