
A better way to rate films - bootload
http://blog.goodfil.ms/blog/2011/10/07/a-better-way-to-rate-films/
======
wvl
I really disagree with the premise of this post. Emotional impact (United 93),
psychological impact (Black Swan), intellectual demands (Primer) all equate to
a lower rewatchability score, since watching the movie takes a lot out of me.
However, the more of an impact a movie makes, the _higher_ my opinion of a
movie. Because something is light and fluffy doesn't make that movie "better",
it just means it's easier to watch.

Rating films this way may do a good job of the "so bad it's good" camp (Snakes
on a Plane), but will do an incredible disservice to movies like Black Swan.

~~~
glen_goodfilms
This is a really important point, and it'll probably justify its own blog post
soon, but a quick response would be this:

I'd class United 93 and Black Swan as being, sort of, emotionally exhausting.
And yes, I'm not going to rush out see them again. But that's very different
to saying they're a bad film. I would suggest that, at high levels of quality,
low levels of rewatchability makes it actually a better film, in a way.

Or, at least, it differentiates them from another terrific film like The
King's Speech. If you're the kind of person who likes watching beautifully
constructed but largely enjoyable films, your tastes will tend toward the top-
right of the graph. But, if you enjoy more difficult or harrowing films, your
favourite films might be lower on the rewatchability score, on average.

My point is, while quality is quality, a measure of skill and talent,
rewatchability is personal, and groups similar films together in interesting
ways. Where your favourite films come out on the graph is really not important
to anyone but you, when you're looking for more stuff you like.

You may still disagree, but thanks for taking the time to comment :)

-glen.

~~~
wvl
I guess then, it depends on execution. If you're combining both of these
numbers into one single metric of "goodness", then you'll just be
accomplishing Pahalial's point of "legitimizing 'poor taste'". If, however,
you can expose this information separately and well, then perhaps it will add
value.

Personally, I'd like to break down the "rewatchability" metric into more
components, to really get at the heart of the matter, like others here have
commented. Pace, quality, fun, emotional impact, length, genre -- these are
the things that really matter. But now we're getting into movie geek
territory.

Speaking of movie geek -- I can't believe you left out the most interesting
thing in that blog post. Where can we see a breakdown of what movies are where
in that scatter plot? I want to know what those outliers on the high quality /
low rewatchability scale are!

~~~
glen_goodfilms
Yeah there's no sense in combining them. Or even averaging scores, really.
Using a scatter plot you can get quite a good insight into the film at a
glance.

As for which movies are the outliers, that you'll find out when you're a
member :)

~~~
carbocation
On your scatterplot, the correlation between the two scores looks incredibly
high. By eyeballing it, I'd say > 0.8. Will you give the actual score in your
upcoming post?

Ideally, it would be nice to find two axes that are orthogonal and let people
rate on those. Clearly these two axes are highly correlated.

------
Pahalial
This seems useful, yet awfully much like another step on the slow path to
legitimizing 'poor taste'.

That's a bit harsh, so let me qualify it. I actually rather agree with the
division inherent in this and think it will resound with people, but I
honestly believe that this is yet another step towards turning our society
into idiocracy. It's the slow legitimization of appreciation for content that
we acknowledge as garbage but consume anyway. I firmly believe that there is
inherent social value in this garbage content remaining quasi-taboo - a social
pressure not to like such things. Sure, most people will watch some anyway, or
play the latest mindless shooter, or what-have-you, but so long as it's
understood that most people will mock you for publicly liking this, the norm
does not drift towards mindlessness - you still need to consume some
legitimately good content, even if only to have something to talk about.

I don't mean to attack the site; I think it's a great way to slice into the
recommendations game, and I do think a lot of people will get value out of it,
and it seems like it may well succeed. I simply am afraid it's a net negative
for our society in the long run (then again, of course, not launching is
hardly going to reverse such a trend.)

Edit: Make that _symptomatic_ of a net negative. I've thought some more and
this is only giving an online portal to express the shift I talk about above,
which has already been happening aplenty.

~~~
recursive
I think the alternative is people pretending to like things they do not. You
can't force people to appreciate your brand of high culture by mocking them
for not doing so. At best, the people you're trying to win over will publicly
pay lip service. At worst, it will become more difficult for people like you
to find things that you think are worthwhile, because the space will become
polluted by people who don't get it. Let people like what they like.

I, for one, am not ashamed to say that I like stupid things.

------
InclinedPlane
It's still broken, just differently (maybe better) broken.

Films, music, books, etc. are not about some abstract universal gradation of
quality that a large enough, good enough statistical sampling of viewers will
reveal with increasing precision. No, they are art. Enjoyment and appreciation
of art is a matter of personal taste, which is not, will not, and cannot be
universal for everyone.

That's why up/down votes are decent enough ratings systems despite their
crudeness, all they do is tell people whether or not taking the effort to find
out if a piece of art suits your personal taste is likely to be worthwhile.

Ultimately what ratings are trying to get at is recommendations. The simplest
idea being to measure the average opinion of everyone and present the highest
rated works as recommendations. However, more sophisticated systems which try
to find people with similar taste and recommend to you things that they seem
to like but you haven't been exposed to are likely to be more successful,
though significantly more complex to implement. It seems unlikely that even
that represents the limit of improvement for the problem, but nobody's done
much better yet.

------
seanalltogether
I've thought about this problem a lot and I'm glad to see there is a site out
there trying to tackle this problem. For me, I always thought about movies in
terms of investment with 4 options, "I'd buy it", "I'd rent it", "I'd watch it
for free", "I'd avoid it". I think dollars speak louder then opinions. The
problem is that nowadays in the world of streaming video, this scale gets
completely trashed as the friction to invest in a movie is lowered
considerably.

I do like the idea of production value vs replay value, I remember when I saw
Passion of the Christ I told a friend it was an amazing movie i never want to
see again.

------
ComputerGuru
I posted this on the site, posting it here too:

I would say a better metric would be "How much did you enjoy this film?" vs
"How good is this movie?"

I've been grappling with this problem since subscribing to Netflix, wondering
how I should rate a movie I personally did not enjoy _that I also felt was
incredibly well made_. For example, "A Clockwork Orange" is an incredible
movie by all means and I would give it 97 stars out of a hundred.... but I
hated every single second of it. It's gruesome, sadistic, dark, miserable, and
disgusting. But it was undeniably well made.

Rewatchability sucks in the aspect because many movies can't be rewatched,
while still being both splendidly produced and enjoyable. Let me make this
clearer with examples: "The Sixth Sense," "Rear Window," "The Accused," "The
Diving Bell and the Butterfly," "12 Angry Men," "Apollo 13..." The list goes
on and on. There are great movies that just don't rank so highly because
they're SUSPENSEFUL and that suspense doesn't translate will into rewatches.
Rewatching suspenseful movies will make them seem boring and long-drawn.

Rewatching feel-good movies ("Seven Pounds," "How To Train Your Dragon,"
"Blind Side," etc.) however, is just as good the second time if not better.

tl;dr the rewatchability index will be heavily skewed towards feel-good
favorites and epics, and away from dark, gritty, mysterious, and suspenseful.
Not a good index.

------
wernercolangelo
Discolsure: I am a co-founder for a movie recommendation site called
tellmetwin.com so I am biased towards our own rating system...

I have never considered the rewatchability criteria as a potential factor in
rating a movie, but must admit that is has appeal. But thinking about it
further, it is a very personal metric so would not be very useful in
determining the overall watchability of a movie. I.e. if you consider a movie
rewatchable it might not correlate with me considering it to be watchable.

I would love to be able to see a plot of individuals and how their ratings
cluster (some interesting visual way of adding and removing individuals would
be cool) because I would guess (based on our own data) that these plots would
not be uniform. And if they are not, then an individual's movie
recommendation, based on rewatchability, would have less value to me.

------
socratic
I really want a good movie recommender system. (Which I suppose makes this a
good idea to work on.) However, focusing on these sorts of issues seems like
exactly the wrong way to go about it.

I understand that people overwhelmingly tend towards the extremes of rating
scales. But who cares? There's so much noise anyway.

Here's what I want: (1) include (all) films and television series and (2)
include top 1000 lists for as many niche categories as possible (foreign films
about relationships, teen comedy, and so on).

I don't even care if it is personalized, because the personalization is always
horrible. (I've tried Netflix, Jinni and others and they're all terrible.)
Just having the average for the movie is actually amazingly accurate on many
measures (within 10--20%).

Is there something like this? The closest I've gotten so far is finding a good
IMDB user list or two.

~~~
covercash
While not exactly what you're looking for, I find myself going back to
<http://nanocrowd.com> quite often for movie recommendations.

They don't do TV shows, but their niche categories are usually pretty solid.

It works well if you can give them an example of the type of movie you feel
like watching. For instance, I wanted to watch a cheesy, nerdy, thriller
earlier tonight, something along the lines of Hackers and it suggested The
Net, Antitrust, TRON, Live Free or Die Hard, The Matrix, Swordfish, etc.
Exactly what I was in the mood for.

------
phzbOx
Interesting post even though I'd admit that I'm not a huge fan of "I'd watch
it again". For instance, some movies are really _hard_ (huge drama, etc.), or
particularly long, or even have a huge punch at the end. These movies would
have a "rewatchability" really poor.

One might also watch a very simple movie.. nothing extraordinary that you
wouldn't necessarily watch again; that doesn't mean it's not a good movie to
watch for the first time.

Personally, I think the best factor to "guess" if I would watch a movie would
be to anonymously analyse a huge set of data and find people who like the same
stuff as I do. So, he likes x,y,z (as I do) and hate a,b,c as I do, good.. we
mostly like the same stuff. Now, he _really_ enjoyed K; I'd probably also like
it.

~~~
phzbOx
(Note: This is different than what Amazon does with the "People who bought
this item also bought these items".)

------
dexy
I think rewatchability is a great metric for one thing: whether or not someone
is going to want to rewatch something...

Just because the data better fits your desired skews, doesn't mean it gets at
what the users actually want.

I think a major issue at hand is that people often rate things as they expect
a critic to rate them. If you would give Transformers 2 a 5-star
rewatchability rating, you should be giving it at least 3 or 4 stars for
quality, because obviously you enjoyed it very much. Instead, many people
pretend to be film critics, whose jobs are very different, and assign an
'objective rating'. A film critic can not give personal opinions because he's
supposed to speak for the masses--to appeal to some higher taste he aspires to
have and that he hopes society would have. If you loved Transformers, rate it
5 stars--period. The rating isn't meant to be read by others, it's not meant
to appeal to an idealized world, it's just how you felt about the thing.
Convince your users to rate intelligently with that mindset and you'll start
to see good data.

All that said, more data is better than less, and if you could convince your
users to double their average rating-time investment and give you ratings for
both, awesome. I'm skeptical that users would bother to rate two metrics for
everything they've seen...but I think it's certainly possible they would. Be
interested to see.

------
tellmetwin
I like the term "rewatchability" but I don't agree with your logic.

Why would you limit your pool of advice-givers to your friends?

They are good for service advice where you need someone you can trust to give
you the whole and honest story.

In movies it's beyond that, it's much more important to find people with the
closest match to you in movie taste as possible. The more people the closer
taste match you will get (goes without saying).

I also don't see the point of giving "quality" scores. I really don't care if
a stranger thinks some movie was of high or low quality. I'd much rather know
if my taste-twin loved it or hated it. I for example thought Lord of the Rings
was super well done and yet super boring. But this opinion of mine only
matters to someone who resembles me in movie taste, who hates and loves the
same quirky things as I do about movies. And of the two opinions i mentioned
(well done, boring) its really only the latter that matters to my twin, he/she
should pay most attention to that when deciding if to go and see the Lord, not
if I thought the movie was of high or low quality.

But having said that, I like "rewatchability" as a concept to study further. I
too would rip of my nails before seeing Black Swan again but I still thought
it was a masterpiece. However, I have seen Requim for a Dream (Darren's
Arrenovsky's previous movie) again and again although it was also very
emotionally intense like the Swan. Just in a different way, less depressive,
but killer strong in an exhaustive yet positive way. And then.. yet and
still.. I would recommend both movies to everyone I meet. I think both are a
must see even though one of them I would never want to see again.

------
ph0rque
One thing that movie ratings don't take into account, including this one, is:
what am I in the mood for? I'd like to type into netflix, "I'm in the mood for
James Bond meets Spirited Away", have it perhaps give me a few yes/no choices,
and make recommendations based on that.

~~~
socratic
This is the main thing Jinni claims to do.

Have you tried it?

<http://www.jinni.com/>

~~~
daeken
I also strongly recommend <http://tastekid.com/> . The recommendations are
great, although it tends to give them very narrowly (very similar movies,
rather than related movies you might enjoy but are in a different style). The
name is also very unfortunate, but I love it.

------
mkolodny
I think that "rewatchablility" is an important metric, but it's just one of so
many different metrics that people use to judge what a great movie is.

As you said, there are plenty of terrific movies for which I wouldn't be upset
if I never saw them again. Yet, would I recommend them? Yes.

Movies are so wonderfully complex that there is no single metric by which we
can judge their credibility. We can only state our opinion of them, and leave
the judgement to the next viewer.

------
nostromo
Another issue with five star ratings is that visually you would suspect that a
50% rating is 2.5 out of 5 stars.

However, the 50% rating is actually 3 stars. This is obvious when you list out
your star options: 1,2,3,4,5. 3/5 stars visually appears to be better than
50%, but is actually exactly 50% due to the inability to choose 0 stars.

(This seems unresolvable as most people now recognize the image of 5 unfilled
stars as null rather than 0%.)

------
pbreit
Surprisingly doesn't mention IMDB which has by far the best general purpose
ratings.

Curious if they considered "Would you watch this movie again? [yes/no]"

I also wonder if NetPromoter could be adapted for rating things like movies.
It uses the question "Would you recommend..." and has a much harsher formula
(+1 for 9/10, -1 for 0-6).

------
jonkelly
I definitely liked the personality that came through the writing. "On the
other end of the scale, I think Black Swan is a terrific film, but I’d sooner
pull my nails out than watch it again." - well said and a good illustration of
the value of the dual scale.

------
rickmb
This is purely a way to rate movies for entertainment value. Which means
although the method may be better, the end result is very likely to barely
differ from the current abysmal state of IMDB ratings.

------
cpeterso
The graph of film "quality" versus "rewatchability" looks pretty linear to me.
Why maintain two rating criteria if data shows they are practically correlated
1:1?

------
billmcneale
Any rating system works great if all the ratings come from people with similar
tastes as yours.

~~~
tellmetwin
spot on bill, in a nutshell.

