

How wise are crowds? - Naushad
http://newsoffice.mit.edu/2010/crowd-wisdom-1115

======
chollida1
If you want a real world example of this.

There is a phenomenon in finance called Post Earnings Announcement Drift
([http://en.wikipedia.org/wiki/Post-earnings-
announcement_drif...](http://en.wikipedia.org/wiki/Post-earnings-
announcement_drift)), meaning there is a tendency for companies to move too
much or too little after announcing their earnings and their price eventually
snaps back to a more suitable price.

I spent the better part of 3 months trying all sorts of models to figure out
how to trade this.

The most profitable model I could come up with....

Take each analysts who follows the company, rank them by their previous
historical success at predicting earnings and then weight each analysts'
estimate by the analysts rank.

This grade 8 level math beat out my "fancy" regression models.

I was pretty humbled by that discovery:)

Hopefully my blunder can save someone else here 3 months of their life:)

 __The only caveat to this is, beware of crowds when guessing costs nothing.
When you get free guesses then the results can become useless very quickly. __

The analysts rankings work as a financial analyst has exactly one currency,
their reputation for predicting earnings. When this goes, so does their job:)

~~~
infinite8s
What do you think of Estimize (crowd sourced estimation)?

------
tessierashpool
I just want to start off any discussion of this with an accurate understanding
of Surowiecki's catchphrase.

"The wisdom of crowds" doesn't refer to a general tendency for large groups of
people to be right about anything, under any circumstances. It refers to the
fact that crowds with market-like dynamics tend to be surprisingly good at
ascertaining information accurately.

It suggests that the whole converging-on-the-right-price aspect of markets
might be a specific instance of a more general phenomenon. For instance,
gamblers are also quite good at assessing probabilities, and when country
fairs hold competitions to see who can guess the weight of a cow, the guesses
tend to cluster around the correct answer.

Obviously, as with anything related to economics, there are a ton of caveats,
and people who don't understand the phrase's intent have misused it wildly.

~~~
baddox
> For instance, gamblers are also quite good at assessing probabilities, and
> when country fairs hold competitions to see who can guess the weight of a
> cow, the guesses tend to cluster around the correct answer.

Does each guess submission cost money? If so, then that's a pretty standard
market. People who know more about cow weights are more likely to spend money
because they have more to gain. Of course, even with free tickets, it could
just be that people at rural fairs know a reasonable amount about cow weights.

~~~
ghaff
Under the right circumstances, prediction markets can work surprisingly well
even when no money is involved and even when not all the participants are
expert. Of course, there needs to be some expertise/information in the group.
Random picks aren't going to converge on the right answer.

------
alexashka
What are they saying? Can somebody explain like I'm 5?

"the idea that aggregating or averaging the imperfect, distributed knowledge
of a large group of people can often yield better information than canvassing
expert opinion."

What? Often? If I have a pain in my lower right abdomen and I poll the human
population, I'm going to get a better answer than a doctor?

If I poll the human population about a physics problem, I'm going to get a
better answer than a physicist?

"But as Surowiecki himself, and many commentators on his book, have pointed
out, circumstances can conspire to undermine the wisdom of crowds. In
particular, if a handful of people in a population exert an excessive
influence on those around them, a “herding” instinct can kick in, and people
will rally around an idea that could turn out to be wrong."

Conspire? Undermine the wisdom? Wow...

So when somebody realized blood-letting wasn't helping, I guess those folk
'conspired' to 'exert an excessive influence on those around them' until a
'herding instinct' kicked in...

 _Brain explodes from the stupidity_

When it's wrong, somebody conspired, when it's right, oh, well, don't mention
that it's the exact same process.

~~~
ghaff
As tessierashpool notes, it's not a statement that a random bunch of people
will always give a better answer than an expert, especially if that expert
knows the correct answer with high confidence. Rather, it's an observation
that crowds who know something about the question being asked will often do
better than experts who are also "guessing."

It requires a particular set of circumstances and tends to be most associated
with people independently guessing numbers about things they can hazard an
intelligent guess about.

~~~
alexashka
"It's an observation that crowds who know something about the question being
asked will often do better than experts who are also 'guessing.'"

Can you give some examples of this? I don't think that works at all. If you
ask me and 100 strangers about programming, I'll do better than 100 strangers
every time. Unless one of them is a better programmer than me and has the
authority to convince the others that they should listen to him/her OR! they
work together to solve the problem. In which case um yeah.

So either way it boils down to 2 groups of experts, not the crowd.

The 'particular set of circumstances' are not well defined in the article from
what I can tell - which renders the whole thing worthless doesn't it?

It's like saying it is going to rain. When? Well, under a particular set of
circumstances it will. lol :)

When it comes to guessing numbers - there are no experts, so who is the
crowd's opinion better than? I still feel like I'm 5 and not getting it.

~~~
ghaff
I'll give you one very specific example from data that I crunched myself.

A friend of mine has run an Oscar contest for about 25 years.

Everybody picks their choices in the usual manner. In addition, he has a score
for "consensus"\--which is a virtual contestant which predicts each category
winner based on the most popular pick. For example, if 30 people picked
Birdman and 20 picked Boyhood, consensus would pick Birdman.

Here's the thing. A few people usually beat consensus in a given year.
However, over the course of 25 years, no single person has a better average
score than consensus and, in fact, not even a small panel of those who
(retrospectively) were the best pickers do.

As others have said, in contests like how many jellybeans are in a jar, the
median or mean tends to do better than the vast majority of individuals as
well.

------
mrep
If I remember correctly, the book also described another phenomenon where if
the guessers were allowed to see everyone else's guesses, there was a good
chance that everyone else would end up guessing closer to what the first few
people guessed. This decreased the accuracy of the crowd because it placed
higher weight on the first couple guesses and removed most of the variance in
the guesses resulting in decreased accuracy.

------
bko
My finance professor told my class a story about his wife and himself were at
a fair and encountered one of those games where you have to guess the number
of jelly beans in a jar. His wife has a consulting background while he has a
background in finance. She observed the jar, tried counting the number of
jelly beans on the bottom and extrapolating an estimate from those values. He,
on the other hand, just looked through the first couple of cards others had
turned in and picked a number that seemed in line with the estimations of
others.

He got closer to the actual value.

The wisdom of crowds approach is alive and well in finance. For better or
worse, the wisdom of the crowds approach allows an analysis with much less
work than the engineering approach. But more importantly, it serves as a type
of defense mechanism to the person making the actual prediction especially if
he has no skin in the game. If you're way off, at least you're in good
company. Okay for counting jelly beans, but it has a potential of being
disastrous in other situations.

------
simonswords82
Well that's a relief. I only read that article as I've usually relied on a
"wisdom of the crowds" approach to large decisions. I tend to reach out to a
number of people inside and outside of my immediate "herd" and then aggregate
their opinions to help influence my own.

Had I just learned that the wisdom of the crowds was not a thing I would have
had to seriously re-evaluate my life!

------
napowitzu
> For instance, he says, if you notice that a Chinese restaurant in your
> neighborhood is always half-empty, and a nearby Indian restaurant is always
> crowded, then information about what percentages of people prefer Chinese or
> Indian food will tell you which restaurant, if either, is of above-average
> or below-average quality.

The situation described here wouldn't tell you anything about the quality of
the restaurants. It _might_ tell you what kind of food that neighborhood
prefers. It _might_ tell you which restaurant is more affordable. It _might_
tell you which restaurant has a better public image. Or maybe it just tells
you which restaurant is renting out its parking lot to a nearby car repair
shop. To wit, quality cannot be determined by observing human behavior.
Quality is determined exclusively by predefined parameters. Loudness,
brightness, velocity, purity, etc., are measures of quality. In other words,
quality is antipodal to popularity. Every human relation to an object, such as
cost, availability, relevance, etc., has priority over quality at all times.
It is even more accurate to say that quality is never a direct factor in
choice. A person does not choose a knife, for example, because it is sharper
than other knives; a person chooses the knife because it is _sharp enough_ to
meet their needs and, more importantly, because it is accessible to them. The
only exception is the case where a person seeks out the highest quality option
for the sake of quality itself, i.e. "I want to find the sharpest knife in the
world." In this exceptional scenario, however, one learns nothing by studying
the quality-seeking crowd that could not have been learned by simply measuring
and comparing objects, i.e. it is much easier to find the sharpest knife by
measuring knives for sharpness than by trying to determine which humans buy
knives strictly because they are sharper than the alternatives (in such a case
you would still have to measure the knives to confirm any conclusions you came
to). All of this being said, I can't believe people at MIT studying human
behavior would make such an absurd analogy. It indicates a complete
misunderstanding of not only human beings but of logic as well.

~~~
sukilot
Yes, in theory, theory is more accurate that practice.

------
alwaysdoit
However, this model assumes that the people are actually sincerely trying to
help you. In practice, any use of the wisdom of crowds really boils down to
dealing with abuse/trolls/spam.

------
beefman
Original paper (since the link in the story is broken):

[http://pages.stern.nyu.edu/~ilobel/bayesian-learning-
social-...](http://pages.stern.nyu.edu/~ilobel/bayesian-learning-social-
networks.pdf)

~~~
aet
I see this paper is 8 years old..

~~~
beefman
The story is from 2010, the same year the paper was finally published.

------
seeingfurther
Does anyone remember the name of the wise crowd app that just launched in
beta? It's a GUI interface that let's groups pick things like basketball game
results or the oscars?

------
aet
Does this article contain the name of the paper?

