
Ignoring the Wisdom of Crowds - joeyespo
http://blog.asmartbear.com/ignoring-the-wisdom-of-crowds.html
======
freshhawk
I stopped taking this seriously when they compared the accuracy of the "ask
the audience" and "phone a friend" bits on Who Wants to be a Millionaire
without considering that everyone used the ask the audience lifeline on an
early question and saved the phone a friend for later questions.

Without controlling for when they are used the comparison of accuracy is
literally meaningless.

The same stats show that, rightly or wrongly, the contestants didn't have as
much faith in the audience for later stage, much harder questions.

I guess I should give the benefit of the doubt and assume the writer didn't
notice this and the error is accidental but that's only slightly better than
just ignoring it because you wanted a sexy anecdote to support your point.

~~~
hammock
You right that the two lifelines are used at differing stages in the game. I
wonder though, what the accuracy rates look like after controlling for what
stage of the game you are in. Maybe people SHOULD save their ask the audience
for later in the game.

~~~
T-hawk
Another variable is the subject matter of the question. Anecdotally, it feels
that the audience almost invariably nails the answer by a wide margin on pop
culture questions. On questions of a scientific or technical nature, the
audience opinion is much less clear and more often actually wrong. Phone those
questions to a domain knowledge expert.

About the worst thing you can do, but contestants love doing this, is to burn
the valuable audience lifeline early on a question when you already have a
good sense of the answer but just want to "make sure". What a waste! Save that
for a question when you have no clue, which WILL happen before you reach the
million level.

~~~
nisse72
Something else they often do is say something like "I'm thinking it's B, but
I'm not sure so I'll ask the audience". Well guess what the audience guesses
after hearing that?

------
billswift
His further reading list includes Surowiecki's _The Wisdom of Crowds_ , but
his post doesn't look like he read it himself. Surowiecki has a long list of
requirements for the "wisdom of crowds" to actually be helpful, for one
example the individual judgments need to be independent, that the post
ignores. Without the restrictions you get different failure modes, like
bubbles, runs, and traffic jams.

~~~
smartbear
The essay was intended to be a light-hearted and readable intro to the
concept, not an exhaustive definition that takes an entire book!

~~~
billswift
Almost everyone has at least heard the phrase "the wisdom of crowds" by now.
The most important part of the idea, and the one most people are not familiar
with, are the (fairly severe) limits on its usefulness. I wrote a review of
the book for Amazon, and one thing I pointed out is that it was a much better
balanced treatment of the subject than I had expected from all the hype about
the idea that I had previously read. The hype is already all over the place,
and has been for years, there is no need to add to it.

------
smoyer
As programmers we should be well aware that our customers' "needs" are bounded
by what they a) know and b) believe is possible. Guiding them to technological
enlightenment can and does provide rewards (financial and otherwise).

~~~
fleitz
Are you essentially saying that if you listen to your customers you'll build a
faster horse, instead of a car?

~~~
smoyer
No ... I'm saying that sometimes we are evangelists and can make them realize
they want a flying car.

------
antithesis
He isn't being very fair: where the goal at first was to be as right as
possible, it later became "to piss off as few of the 300 guests as possible".
I'm not sure what exact procedure he had in mind for businesses, but in a
situation where "votes eliminate the interesting edges, leaving only the
boring residue that no one hated enough to vote off the island" applies,
you're just not talking about the wisdom of crowds anymore, this wasn't
possible in his earlier examples either. An important factor to the wisdom of
crowds, and a key difference between his earlier examples and his later
examples, is that people vote independently, without even talking about what
they're going to vote.

For demonstrational purposes, I will now take his holiday meal situation and
apply actual wisdom of crowds to them. This isn't a very usual situation, but
this is about properly applying the wisdom of crowds.

Instead of people stating what they dislike, people may nominate a meal, and
afterwards, everyone votes for the meal they want to go with. Like in the
Wants to be a Millionaire examples, people cannot talk in this process - they
cannot state what they dislike. Instead, the fact that some people can't take
spicy food will show in the results, but beside that, every individual will
pick a meal that isn't boring, and the end result will be just that. "Allergic
to garlic" and "doesn’t eat anything green" are rather specific complaints and
were not taken into account, so let's count those 3 people as unsatisfied, but
the success here is that most people will in fact be satisfied with the meal.

~~~
smartbear
You can't use your technique for meals because everyone will suggest a
different meal, and then you have no basis for a decision. That technique
works when you're voting on one variable which has one CORRECT value which
you're trying to seek. Creative work is not that.

------
wisty
The other problem with focus groups - they aren't independent. There will
always be one person who sounds convincing, and the rest will be biased to
whatever that person thinks.

The advantage of focus groups is, you can identify themes which resonate. For
example, the group might say "integrated" sounds good. That doesn't mean you
want an integrated product, it just means that you need a sticker which says
"integrated", and some justification (We integrate with Facebook, via our
"Like" buttons!)

~~~
hammock
>The other problem with focus groups - they aren't independent. There will
always be one person who sounds convincing, and the rest will be biased to
whatever that person thinks.

Experienced group moderators are aware of this take steps to mitigate and
reflect in their notes.

There are other forms of market research beyond focus groups. Aggregated
quantitative and qualitative one-on-ones also fit the OP's bill of "crowd
wisdom"

------
ozim
This essey is weak, I mean author is comparing apples to oranges. First
problem with jellies is Fermi problem, it is approximation. If you have more
people giving they answers mean of approximation will be closer to actual
value. But second part is about concrete thing it is not estimating or
guessing. First part is totally irrelevant. Second part could be used alone.
But you just know that you can't please everyone so it is common sense I
think. Like going for trip with people who are working in different jobs on
different schedules, somebody will have to be left out.

