
The candy weighing demonstration, or, the unwisdom of crowds - BCM43
http://andrewgelman.com/2008/05/08/doing_the_candy/
======
baddox
> Now call out to the students who are sitting near where you hid the
> envelope: “Um, uh, what’s that over there . . . is it an envelope??? Really?
> What’s inside? Could you open it up?” A student opens it and reads out
> what’s written on the sheet inside: “Your guesses are all too high!”

Maybe it's because I recently did some reading about magicians, but if I were
one of the students I would be thinking that he could have any number of
hidden envelopes with different predictions, and he just chose the one that
ended up being correct. Of course, I'm deliberately missing the point of the
story.

~~~
sp332
The point is that they all guessed low. The weight of the bag never changes,
and there's no pretense that the teacher didn't know how much the bag weighed
all along. You don't even need the envelope - you could just weigh the bag at
the end. Same effect.

------
ska
The "wisdom of crowds" part of the title is a bit unfortunate. While there can
be systemic problems with that type of approach, this does not really
demonstrate them.

What is demonstrated is when you give the students an algorithm for a biased
estimator, the estimate they get is biased. This is good; empirical
demonstration is useful... but it isn't the wisdom/unwisdom of crowds, really.

edit: good responses! Unfortunately I don't have enough time right now to
properly clarify how/why I'm looking at it this way.

~~~
wpietri
Seems fair to me. Note that he published all the estimates and then had people
make predictions as a group, which is pretty much the way to extract wisdom-
of-crowds effects.

I agree the estimator is biased, but in good part because of how people pick 5
from 100.

We are all biased in some ways. If a group shares the same biases, then a
wisdom-of-crowds approach will yield a wrong answer in which everybody is
confident. I think it's a fine lesson for people trying to work in this
fashion.

~~~
sp332
He didn't have them estimate anything. They weighed things with a scale, then
multiplied by 20.

~~~
ska
But they shouldn't all be doing the "same" thing!

~~~
sp332
Hm... So what was the point of writing all those steps on the board?

~~~
wpietri
To constrain variation to how they picked the candy.

------
ChuckMcM
In this day and age nobody is going to hand the scale to the next group with
the bag sitting on top of it? Its a bit cynical but I would expect that some
pairs estimate to be really really close if not spot on as an effect of they
just weighed the bag directly.

~~~
abcd_f
Not even that.

As you pass the 1.5 kg bag to the next group, it's easy to see that that it's
_not_ 3-4 kilos as you have just guessed.

~~~
pgeorgi
People routinely overestimate weights. Speaking from experience, it's possible
to routinely win bets with an unmarked 2kg dumbbell (which weights "at least
5!")

------
lotsofcows
People forget that Galton's original example was with a group of people with
good domain knowledge. It would be interesting to try the experiment on a
group of old-school sweet sellers.

------
bentcorner
I wonder if he just asked the students how much the bag weighed without giving
a scale if you'd get a better answer.

[http://phenomena.nationalgeographic.com/2013/01/31/the-
real-...](http://phenomena.nationalgeographic.com/2013/01/31/the-real-wisdom-
of-the-crowds/)

This post is more "ask a bad question, get a bad answer".

~~~
npsimons
This post is also from a teacher of statistics. Sure, you can weigh the whole
bag to get the total (or average) weight; how do you propose to measure an
aspect of an entire population, say one as large as a country? Sampling has
its uses, when done properly. Besides, as others have pointed out, most people
are notoriously bad at estimating weight.

------
chime
When you shake the bag, smaller candies settle down to the bottom, the larger
ones get to the top. So even if someone tries to shake the bag to get a
'random' sample, they will be getting a biased sample if they pick all 5
candies from the same layer (usually the top).

~~~
jckt
There's a name for that.

[http://en.wikipedia.org/wiki/Granular_convection](http://en.wikipedia.org/wiki/Granular_convection)

~~~
huhtenberg
Yup.

That's how you increase fruit content in your müsli - shake the bag, scoop
from the top and leave oats to others :)

------
secondForty
Neat example.

So nobody in the class just waits until you're not looking and weighs the
whole bag with the scale?

~~~
bittercynic
You could make that strategy difficult by using a big, floppy bag and a scale
with a small platform.

Or simply a scale incapable of measuring a weight much larger than the
heaviest piece of candy.

~~~
mturmon
Making the bag itself heavy would also help.

------
calinet6
A wonderful demonstration of statistics.

On similar ground, this reminds me of Deming's red bead demonstration,
relating statistics to corporations and management practices. Best explained
dynamically:
[https://www.youtube.com/watch?v=JeWTD-0BRS4](https://www.youtube.com/watch?v=JeWTD-0BRS4)
(delightfully, this is posted by the Mayo Clinic).

~~~
Patient0
Thanks for sharing this. Initially I found the video to be a bit dull but on
reflection I think it's a brilliant demonstration on how companies tend to be
managed in practise.

~~~
calinet6
Exactly. Look into more of Deming if you liked that; he was brilliant all
around and accurately evaluated and solved many management problems on a grand
scale.

------
mappum
The real reason for the bias is that everyone thought they could sneak a
piece.

------
ZoF
Kind of surprised that kids don't realize this.... What grade is it in?

The first reaction I think any class I was in would have to this demonstration
would be to figure out how we're being cheated.

Given a bag with a random sampling of candy and being told to 'pick 5 pieces'
I doubt I would choose 5 of the same large candy bars.

It seems highly surprising that 100% of the time this is done you don't have a
single pair of students reaching just a bit further into that bag.

It doesn't even seem to be a particularly impressive demonstration.

~~~
phreeza
By that logic, shouldn't they be guessing too low? I think the point is they
realise there are different sizes, but consistently overcompensate for this
effect.

------
Robadob
I would really have expected atleast a small number of people to have
carefully looked to workout the distribution and whether there were a higher
density of one type (e.g. small) at one position.

I imagine the use of a bag rather than a Jar as per the usual school fair game
could make spotting these harder though (unless they're allowed to pick up the
bag).

------
the_cat_kittles
this isnt the unwisdom of crowds, its the difficulty of noticing sampling bias

~~~
gwern
Correlations between participants (such as due to a systematic bias such as a
selection bias) is one of the most common ways for 'wisdom of crowds' to fail.
So I'm not sure what your point is.

~~~
the_cat_kittles
My point is this accidental selection bias doesn't depend on the crowd- you
could just as easily make this mistake if you are 1 person in a room by
yourself. Some examples of what I would call unwisdom of crowds might be the
biasing feedback effect a prediction market's current price has on people's
judgement, or how those with the most money (or the with the smallest marginal
value of a dollar) exert a larger influence on a prediction market's
predictions, if they are implemented that way.

------
stuartd
I thought 'aftermath' was a great pun (HN must be Americanising me because of
course it should be 'aftermaths')

------
Houshalter
Maybe it just weighs less because some of the students are secretly eating the
candy.

