
Analysis of whether IMDB ratings suffer recency bias - eoinmurray92
https://kyso.io/KyleOS/movies
======
8innovate
This is definitely not the 'recency bias' I expected.

I've always noticed that when a show is released it gets WAY higher ratings in
the first week or so of the release date. I've learned not to trust the
ratings until at least a 2 weeks after a release.

It looks like this analysis focuses on the year a film was released. This
seems like it would be a lot harder to determine because film quality changes
(subjective) and the audience writing reviews is probably changing over time.

------
PaulHoule
It's well known that movie rankings do. This was key to the resolution of

[https://en.wikipedia.org/wiki/Netflix_Prize](https://en.wikipedia.org/wiki/Netflix_Prize)

The time series of reviews of movies like

[https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film)](https://en.wikipedia.org/wiki/Sucker_Punch_\(2011_film\))

and

[https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus...](https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Justice)

tend to have many early bad reviews by mainstream people who can't get past
things that they find squicky and the lack of the usual "Hollywood Happy
Ending" but that get better reviews from "true fans" later on who enjoy the
fact that the directors tried to make something different even if it was
flawed.

~~~
slyall
A quick Google shows a few sites that keep history of IMDB ratings.
Unfortunately they seem to concentrate on movies actually in the IMDB 250

Some interesting ones:

Avatar -
[http://top250.info/movie/?0499549](http://top250.info/movie/?0499549)

Se7en - [http://top250.info/movie/?0114369](http://top250.info/movie/?0114369)

Heat - [http://top250.info/movie/?0113277](http://top250.info/movie/?0113277)

Star Wars: Episode VII - The Force Awakens
[http://top250.info/movie/?2488496](http://top250.info/movie/?2488496)

Terminator 2 -
[http://top250.info/movie/?0103064](http://top250.info/movie/?0103064)

Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012,
probably due to a change in the ranking formula.

------
minimaxir
Recency bias came up a few days ago in a previous HN submission by the OP, and
the OP also commented in:
[https://news.ycombinator.com/item?id=20099023](https://news.ycombinator.com/item?id=20099023)

In that subthread, I posted a heat map data visualization I made of IMDB
rating vs. release year, which is suspiciously similar to the one OP made:
[https://news.ycombinator.com/item?id=20099344](https://news.ycombinator.com/item?id=20099344)

It's not plagiarism, but if the OP got the idea for checking for recency bias
from the HN thread, they should explicitly credit that.

~~~
KyleOS
Hey - not OP but I wrote the article. There was a lot of discussion about
recency bias in that thread, including the graph you shared and I decided to
expand on the analysis. Note there are a few other graphs in my post. I did
just draw this up quickly yesterday. But you're right, sorry - I should have
cited your comment in my post. Adding it now.

~~~
bscphil
Hi, there are what seem to me (not a statistician) to be a number of serious
errors in the article. The most egregious is the first conclusion, which
threatens to undermine the whole thing.

> There does not seem to be any strong connection between number of votes and
> a movie's IMDB rating.

Well, this just isn't true. I happen to know this because I've looked at this
data before myself. The issue is that you're cutting off the graph at a
maximum of 60k votes with no explanation or even pointing out that you're
doing it! I'm sure this is just an honest mistake, but it cuts out basically
every movie that is actually popular!

This slowly ate at me until it was enough to get me out of bed to redo this
graph myself. I've uploaded my quick and dirty result in R here:
[https://i.imgur.com/TTuCFEL.png](https://i.imgur.com/TTuCFEL.png)

As you can see from the graph, there's a direct and obvious link between the
number of votes and the average rating. The reason I say this threatens to
undermine the whole thing is that I also know from previous experience that
more recent films tend to have a lot more ratings on average. This (naively)
suggests to me that we should see some recency bias if only because more
recent blockbusters get a much larger number of votes than others and the sort
of people who vote on blockbusters have different standards for what makes a
great movie. (Avengers: Endgame was briefly the #1 movie of all time on the
top 250 list.)

I don't have time to plot all the different graphs you did, but perhaps you
can recheck your results after fixing at least this issue, and get back to us?

~~~
bscphil
Follow up: I redid one of the graphs to satisfy my own interest in the
question. Instead of averaging over all the movies ever released like the
article did, I averaged over all the _votes_. This was an attempt to answer
the question of whether the average viewing experience of a film from year x
is better or worse than that of a film from year y.

I found that there has been a noticeable decline in the average rating of over
half a point (out of 10) since ~1930 or so.
[https://i.imgur.com/bY9vPvk.png](https://i.imgur.com/bY9vPvk.png)

I think this should be explained in a combination of two ways:

* History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good.

* To some extent, older movies may really be better than more recent ones.

I still suspect that the blockbuster effect means that the movies people are
mostly to watch are going to receive higher ratings than the average film
overall. And this is mostly because people rating blockbusters are less
critical than the film-viewing community as a whole. So while there might be
no "recency bias" of the kind this article was looking for, there might be
blockbuster bias where in the last several decades studios have figured out
how to capitalize on a less critical / cynical audience. (That hockey stick
graph of 8-10 ratings in the article is certainly suggestive.)

~~~
bscphil
Actually -- one more follow up for the zero people who will read this --
there's a simpler explanation, which other people who do these kinds of
statistics should probably note.

The average rating between 1920 and 1980 fluctuated at about 7.5. Then the
number of movies exploded, and the variability in quality of them exploded
even more than that. Movies that were _better_ than the 1980 average had their
ratings compressed (you can only do a little better than 7.5 on IMDB). Movies
that were worse were much less compressed (you can easily do much worse than a
7.5). So the average gets dragged down since there are now both more movies
getting 0-4s and movies getting 8-10s.

I realized this after plotting the median, not the mean, and observing that it
stays remarkably constant, and possibly even shows a little recency bias for
the last 2 years.

------
olalonde
Doesn't this analysis rest on the assumption that movies are equally good each
year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure
this is a good assumption.

~~~
jdsully
The problem with past movies is you only see the good ones. I've spent some
time watching the bottom barrel movies from the past and they seem even worse
than Today's movies - because while both suffer from horrible plots usually
modern movies at least have interesting visuals.

However for certain categories the past was most definitely better. Dramas
being the main one. Without a lot of fancy effects the plot was the only thing
you had to carry the movie. They also don't beat things over your head and can
embrace subtlety in a way modern movies do not.

I also really liked the 70s trend of movies that didn't really have a
satisfying ending, you just get to experience the lives of people who don't
really improve. 5 easy pieces or The Last Detail fit that category very well.

~~~
EugeneOZ
So you want to say that Shutter Island is a bad example of drama? It's all
just old-farts rants, nothing more.

Everything is evolving.

In old movies actors play often too dumb, just few of them have good examples
of how to play a role.

Some "legendary" movies from the past would fail in theaters nowadays without
their legendary reputation.

We have more movies right now, more choices, so our demands are higher -
otherwise we will not have enough time to watch them all. Also, we have much
more choices of how to watch them - age of digital distribution.

Higher demands give higher quality, movies evolve both in visuals and in
acting art.

~~~
thegabriele
" Higher demands give higher quality "

I would suggest an higher standard deviation on quality, meaning that you
could reach very top notch standards (your point) meanwhile mass producing
total flat works as in music and i presume, all of the arts (my view).

~~~
EugeneOZ
Lol, compare modern music with trash we had in 80s, 90s. Even nowadays' pop
trash sounds significantly better than old hits made on synthesizers.
Everything evolves, as I said. Maybe you just forgot low-quality movies of
previous years, but they did exist, tones of them.

We only want to remember good things.

~~~
thegabriele
There were no meaning to mass produce and diffuse music in those decades like
there are today. I'm not saying 80s and 90s music was perfect (why pick those
decades?), i say the more the demand the more the money the more the highs and
the lows.

The "quality" you're referring to, i don't know: you're talking about what
features of a sound...fidelity, loudness, dynamic?

Anyway, truly the 80s were the years of trash music...but trash as in metal
though.

~~~
EugeneOZ
Loudness, of course, man. What else is the measure of quality? Loudness,
without any doubts. Of course I talk about loudness.

------
iforgotmypass
I feel like author is missing the point. Shouldn't the recency bias show up in
recent votes (while the movie is "hot" and trendy) vs. later votes?

~~~
eoinmurray92
I think it's been interpreted as recent movies not recent votes.

------
bscphil
Incidentally, in case the creator sees this, this site is borderline unusable
for me on mobile Firefox. Text gets cut off in strange places, I can't zoom
out properly to see all of the animations. "Request desktop site" seems to fix
some things but break others - at least I can zoom all the way out but the
text becomes hard to read.

Situation might be the same on Chrome (didn't check), but regardless it could
use some work since it's pretty broken as is.

~~~
eoinmurray92
Hi - founder of Kyso here, thanks for your feedback we will get to work fixing
those issues now

~~~
bscphil
Thanks! After playing with it a bit it's clear that part of the problem is
that the charts are getting generated on the fly and that's very performance
intensive. My brand new Android phone is just choking on the page, to the
point I can actually feel the phone overheating. Perhaps static images would
be an improvement for mobile devices, with the option to load the full data on
request.

------
djtriptych
I was recently thinking about this as Chernobyl (which was excellent to be
fair) quickly became the highest rated TV drama ever.

~~~
watwut
I mean, if you can suggest other similar shows I will be only glad.

------
ddffre
Of course, there are plenty of examples of that. Like the assassination of the
NK leader movie, which at first received 10 star rating. Now look at it.

Sane goes with all the new blockbuster movies, you should usually wait awhile
for the real ratings to show.

------
dahart
> Number of Movies By Ratings

I’m dying to see all of these as percentages rather than absolute numbers! I
was even thinking that while scrolling down through, and then there the totals
are right at the bottom. @KyleOS are you up for normalizing them?

------
fileyfood500
I see from one of the charts that the average movie rating from 2019 is over
.5 higher than the previous 5 years, in what appears to be a big jump. Is it
possible that recency bias is only relevant on a shorter timescale? It looks
like ratings are higher for movies that are a few months old, based on the
data. I wonder if this is generally true, or if early 2019 happens to have
higher rated movies.

~~~
ac29
Sure - for example, fans of a particular actor, director, or series are more
likely to go see a movie soon after release in theaters as opposed to later
via rental or streaming.

Fans are likely to rate higher than the general public, so over the first,
lets say, 3-12 months, ratings often trend down.

------
in3d
I wonder if recording ratings now and then every month or so would show a
decline in ratings. IMDb must know already.

------
ravenstine
Great job! I'd love to see a similar analysis for Rotten Tomatoes since
there's separate metrics for critics and audience, and I find RT ratings to be
more extreme than IMDb. I'm often surprised to see how many bad movies get a
rating of ~6 on IMDb.

------
IAmEveryone
Series tend to be rated higher than movies, probably because people that don't
enjoy them stop watching and rating after one or a few episodes.

That effect may well be spilling over into movies, considering far more movies
are now sequels.

------
ajhurliman
An alternative way of reading the same data: "Analysis of whether movies are
getting better over time"

~~~
richardhod
The problem with your formulation is that 'better' is perhaps the most
subjective word there is. Define it in measurable terms, and perhaps you have
a reading there. There are many factors to consider which may lead you to find
things less clear-cut, for example here's one or two, with some sub-factors \-
Modern audience Preference for familiar patterns in: (eg) \-- acting styles
\-- music used \-- visual tricks and norms \-- special effects styles \--
acting styles \-- soundtrack loudness \-- colour and high resolution \-- High
Concept movies \-- Fantasy and Comic-related films \-- modern movie themes and
topics \- Internationalisation of movies, so that fans from abroad prefer
modern films which are more universalised and less American-specific (see
literature)

~~~
ajhurliman
Right, I'm not asserting that my alternative title is the correct one, I'm
lampooning the assertion that IMDB is "suffering from recency bias" as opposed
to a more neutral claim like "ratings for more recent movies are higher."

Answering the question of _why_ they went up is a lot harder to prove.

------
z3t4
What I find most amusing is ratings (and also awards/nominations) before a
movie has been released.

------
mynewstartup
In terms of shows, Sopranos has long been dethroned, possibly due to this
effect. Then again, production values will only get better over time, so maybe
the latest shows are of the best quality overall.

Best line:

Fucking Queeahs! (Paulie throws chair at his murder victim spirits)

