
When the 'Wisdom of Crowds' turns on itself: IMDB Edition  - nickb
http://news.cnet.com/8301-13515_3-10000650-26.html
======
wmf
If you allow people to vote after they have seen the rankings, you have the
wisdom of mobs. The only way to get the wisdom of crowds is to ask people
_before_ they have seen other people's opinions.

~~~
Hexstream
I upmodded your post because it already had +9!

~~~
cstejerean
I wonder what would happen on this site if the comments and stories were still
sorted by points but the actual number was hidden (or perhaps only hidden
until after you vote but I don't see why they need to be there period).

~~~
chandler
>> (or perhaps only hidden until after you vote but I don't see why they need
to be there period)

That's an interesting point--why does every site need to quantify for the user
the particular score of a comment? Why not indicate popularity by simply
ordering the comments?

Is it a trust issue?

~~~
eru
You could also give other clues besides ordering - without revealing the
numbers 'in your face'.

~~~
william42
What sort of other clues?

~~~
eru
For example color-shading. While it may still be possible to re-extract
numbers from the colors --- psychological colors may have another effect than
numbers.

~~~
eru
Or perhaps something like Nethack does:
<http://www.steelypips.org/nethack/pray.html#p0>

~~~
william42
That's sort of like what Slashdot does for karma. "Karma: Excellent"

------
iamelgringo
I think that the problem is similar to the eternal September problem. I'm sure
that the people who had been voting and contributing to the ratings on IMDB
had been involved on the site for a while. Now, there's a big influx of people
on IMDB that are voting for their favorite movie. I bet that many of the
people who are voting Dark Knight up have never seen the Godfather or
Shawshank.

Same thing happened to my wife in film school. She went to film school because
she loved the classics of the 30's - 50's. She was stunned to find out that
the kids she was going to school with thought that Austin Powers was the best
film ever.

------
breck
_This mob mentality, brought on by a significant event (in this case, the
release of The Dark Knight), causes a total breakdown in a system that is
otherwise fairly accurate, useful, and trustworthy._

I wouldn't call this a "total breakdown in the system". I would call it a
minor blip. I mean, ultimately the system recommends what movies to watch. If
someone isn't going to watch the Godfather now that it's ranked #3 instead of
#1, then the person is flawed, not the system.

Otherwise, this was an interesting post. I have been astonished that TDK is
#1. Didn't realize people were also downvoting other movies to keep it there.

~~~
william42
The flaw in the system isn't that The Godfather was bumped down. The flaw is
that people are rating the Godfather down to the point at which it changed its
ranking with respect to other movies.

And you're right, it's not a major flaw for a movie ranking system. But let's
suppose something similar to the IMDB system(which would be an approval/range
voting system) is used in an election, and the first place gets President, the
second choice getting Veep. With an election, you don't have real-time
rankings like IMDB, but you have something similar--polls. And that's even
worse, since there's a few points of inaccuracy. Strategically worded polls
could change the results of an election.

~~~
unalone
But that change is a blip in and of itself. Godfather has been bumped down
before. Happened with Lord of the Rings. It climbs back up after the initial
wave.

------
ericb
Wisdom of Crowds was a good book. The sequel, Attention Span of Crowds, was
significantly shorter.

------
pg
Have they ruled out the possibility that the people marketing the movie are
gaming the system? Even if voting was restricted to established IMDB users,
surely a reasonably well connected Hollywood marketing person could round up a
few hundred of those among friends and friends of friends.

~~~
rms
<http://www.imdb.com/title/tt0468569/ratings> is the vote breakdown.

~~~
pg
Hmm, those numbers are definitely too big to be influenced by anyone's
network. Judging from the other top-ranked films, it looks as if IMDB's
userbase is in the middle of changing.

~~~
ChaitanyaSai
I don't think they need to game the system. The lukewarm or negative reviews
on RottenTomatoes are being bombarded with genuine fanboy vitriol. (Many of
the commenters, at least on the first page of comments, have joining dates
that suggest they have been on RT for a while)
[http://www.rottentomatoes.com/m/the_dark_knight/?critic=crea...](http://www.rottentomatoes.com/m/the_dark_knight/?critic=creamcrop#mo)
It is quite amusing in a sad way actually, seeing them trash the likes of
David Denby and Joe Morgenstern.

~~~
unalone
To be fair, those reviews were both fairly terrible. Fanboyism is never
justified, in my opinion, but I agree with the respondants' sentiments.

------
Tichy
Maybe the majority of people would indeed get more entertainment out of "The
Dark Knight" than out of "The Godfather"?

~~~
william42
You're right.

However, if "The Dark Knight" bumping off "The Godfather" was all that was
happening, that wouldn't be a problem.

What's happening, though, is that the release of "The Dark Knight" made "The
Shawshank Redemption" have a higher rank than "The Godfather". This means that
the ranking system has lead to the rating system being used strategically.

The implications of this for approval voting would be an interesting
corollary. The polls could influence themselves.

------
Eliezer
I've seen The Godfather. I've seen The Dark Knight. Now, maybe I'm just
missing the point of that Godfather thing, but I'd say The Godfather had more
dramatic tension and The Dark Knight had more philosophical depth.

No, stop laughing, I'm serious. Tell me what point The Godfather makes.

~~~
snorkel
Leave the gun. Take the cannoli.

~~~
pageman
Michael: Fredo, you're my older brother, and I love you. But don't ever take
sides with anyone against the Family again. Ever.

------
Erwin
Blockbusters popular with the avid Internet users have always started at
incredibly high ratings only to slowly fally to the right place. The LOTR
movies were in top-10 once too.

I suppose IMDB might try to account for the initial voting wave being raving
fans and dampen the initial votes, but why would they? Personally I've
reloaded the "Dark Knight" IMDB page half a dozen times to see if it was still
at #1.

The most popular topic on the IMDB message boards is "This does not deserve to
be in top 250" or "Why is this not in Top 250 why some-other-movie is not?".
People like to keep an eye on rankings and to comment on them.

The main practical use for the ratings IMHO is to get a list of movies within
each genre sorted by the rating, to see if you missed something people thought
was great.

~~~
ragaskar
exactly -- imdb has an internet and populist skew. As long as you take this
into account, their ratings are fine and predictable. Gaming rankings is
practically an official internet pastime, so it's not unexpected that this
event is taking place, although I'm surprised it built up enough steam without
(presumedly) official co-ordination to drop the Godfather a few places -- that
_is_ notable. I'm sure politicians are taking notes as we speak.

------
greenagain
IIRC, the same thing happened at the releases of Spiderman and the Royal
Tenenbaums. Hype does funny things to men.

------
bryanlanders
If you think of this system as a game and in this instance a number of users
have figured out a way to act within the rules of the game to achieve an
outcome inconsistent with IMDB's intended one, then IMDB can simply change the
rules to get what they think is best!

It wouldn't be too difficult to observe the behavior of the offending members
(vote for TDK, within x hours vote down Godfather...) and simply setup
criteria to disallow these actions. You could also set up an algorithm to
detect screwy behavior to alert a developer or block that action for a
duration.

And hey, we throw error messages for repeated button clicks and such so that
click-happy users or spam bots don't take down the system...can't that kind of
precaution be extended if necessary? "We know you're in love with TDK, but do
you really hate The Godfather...come back in a month if you still do!"

~~~
william42
Soon you get websites with people saying "Rate The Godfather 1 before rating
TDK 10."

------
andreyf
That's funny, because the book which coined the term is about exactly this -
how to make Wisdom of the Crowds work.

~~~
wmf
The problem is that people discuss these concepts without reading the book and
thus miss out on the subtleties. The long tail is another common victim.

Disclaimer: I didn't read the book.

~~~
davidw
I'm (slowly) working on the summary here:

[http://www.squeezedbooks.com/book/show/23/the-wisdom-of-
crow...](http://www.squeezedbooks.com/book/show/23/the-wisdom-of-crowds)

Contributions welcome!

------
johnyzee
I don't see why it wouldn't be possible to build into the system some of the
same logic that obviously makes it clear to everyone that there is a problem
with these votes.

A computer might not be able to appreciate the goodness of The Godfather, but
it can be taught that a movie that has consistently been voted great for
decades, does not suddenly fall out of favor in a month.

------
rrf
The problem is not the Dark Knight fans, so much as the algorithms being used
to rank the movies. How can you take advantage of crowd wisdom if you don’t
control for spoofing, spam and mob voting? It’s surprising that there wasn’t
already in place a process for dealing with such a common phenomenon.

------
bprater
Perhaps this is an advantage to Y!Hack not having a downvote button.

~~~
Hexstream
I'm much less worried about that happening here because of the transient
nature of the site. I don't think potential manipulators would find it
worthwhile.

------
attack
The point of top-k rankings is to put items in the proper order. As far as the
crowds are concerned, this is no failure. Sorry your favorite movie lost its
spot.

~~~
bprater
It is fine for "FanBoys" to push Dark Knight into number one. But downvoting
Godfather to put your movie into the #1 spot is plainly wrong.

~~~
unalone
But the Godfather people did it first. Look at the spread of points on The
Dark Knight. Look at the 1s. Those came first.

------
flipbrad
over time it should return to a 'fair' position, once temporary disequilibrium
in the 'crowd' (induced by marketing, hype, whatever) is resolved. It's a
shame people vote 'strategically', not to reflect their true sentiments - but
whatever. Over time, with a diverse enough crowd, 'wisdom of the crowd'
systems are effective and reasonably unbiased.

doubtless IMDB will enjoy the free publicity.

------
MikeCapone
AFter the first weekend, Dark Knight had 9.7 average with almost a hundred
thousand votes. That's just crazy.

~~~
unalone
Nah. 9.7 was with only 3,000. 100,000 only hit a few days ago.

------
snorkel
I'm astonished. The internet hordes usually have impeccable taste in film and
music. "Beverly Hills Chowahwah" is not even in the top 20?!!

------
bprater
Should be easy to cull these votes from the system. Anyone voting Dark Knight
at 10 and Godfather at 1 should get nixxed from the system.

In fact, anyone voting Godfather less than 8 should be nixxed.

~~~
delackner
This reminds me of the recent articles about the netflix optimization
challenge. Voting systems like imdb might be improved with some simple
heuristics that take into account the absurdity of some people's votes.

If a user makes votes that are < 2.0 for a movie that the entire community for
years has rated above 9.0, then probably that user's votes should all be
ignored, because they are totally uninformed.

~~~
william42
Because having unpopular taste is bad?

~~~
eru
Evil.

