
Netflix Recommendations: Beyond the 5 stars - aaronbrethorst
http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html
======
noelwelsh
Good to see the Netflix prize paid off for Netflix. In pure terms of hourly
rate, Netflix managed to get some of the smartest people in the world to work
for less than chump change. In those terms alone it was a huge success, but
the Netflix prize also pushed forward the field so really no-one was
exploited.

The same idea is been commercialised by Kaggle (<http://www.kaggle.com/>) but
there are several issues. Of course there is less up-take as the idea is no
longer novel and the prizes are less. More than that, I think people are
realising that winner-takes-all sucks, and the winning entries tend to combine
so many different techniques that, as Netflix found, putting them into
production is difficult. There is some interesting work on a better model
here: <http://arxiv.org/abs/1111.2664>

~~~
CWuestefeld
The Netflix prize's criteria -- accurately rate a pile of movies -- is a red
herring, though. It may help some, but the broad accuracy comes at the expense
of an optimal algorithm for what's really important.

What I want is a recommendation of what I'll probably like. It is absolutely
irrelevant if mid-range movies sore a 2 or a 3. If Netflix can pick out a list
of movies that I would rate a 5 (and maybe even 4), they've got the holy
grail. Nothing else matters. So why optimize your algorithm to capture the 2s
and 3s as well?

For bonus point, it might be nice to be able to pick out the real dogs. If it
could warn me that I'm about to rent a 1 or 2, that would be cool. But it
doesn't matter if they can tell me which of 1 or 2 it is. The precision is
irrelevant, just tell me I won't like it.

(If I've said this once, I've said it a hundred times. But I guess I'll keep
on like a broken record as long as Netflix keeps trumpeting what an
achievement the Prize's algorithm was.)

~~~
noelwelsh
You're not alone. I don't know if you have a ML background, but it has been
somewhat widely discussed in the community that the way Netflix scored the
competition -- RMSE -- isn't the best way. E.g. <http://hunch.net/?p=949> and
<http://andrewgelman.com/2008/11/netflix_prize_s/>

In the OPs article they mention they monitor if a movie is watched to
completion, which gives them a much better metric to optimise. The other issue
is that this is really a sequential decision making problem. Recommending a
movie has an opportunity cost -- there are other movies you don't recommend --
and the recommendation is an ongoing process, so it is probably best to spend
some time exploring the user's taste on the assumption this will let you make
better recommendations in the future. Accounting for these issues is much
harder in a competition format.

------
underwater
"Of course, when we say “you”, we really mean everyone in your household."
It's interesting that they've made a conscious decision to go that route. My
wife and I share a streaming account which does not allow for separate instant
queues. We have different tastes so I often see prompts to rate or
recommendations to watch films and shows I dislike. If I'm honest in my rating
it seems like I'll skew the recommendation engine away from shows my wife
would like but if I'm not I'll end up seeing only recommendations i done like.
It's a catch-22 situation and makes me distrust pretty much anything that is
recommended.

~~~
novakinblood
I also have kids and that really screws up my "suggested" queue. Hmmm, should
I watch "Inspector Gadget" or "The Girl with the Dragon Tattoo" tonight?

~~~
Terretta
I let a family stay at my home for a month, and forgot my Xbox 360 was on
Netflix. Came back from a trip and the recommendations we're all kids all the
time. Cancelled the Xbox account. Three months later, still all kids, all the
time. Went to website and negatively rated all kids shows (though I like some
things like British lit movies for kids) it suggested, mercilessly. Kept at
that for two weeks. Didn't help. It kept cycling around the dregs of kids
shows, surfacing Pokeman, or old Nik reruns. Found an obscure "don't consider
this" button, cycled through everything again. Five months after that, our top
10 "for you" don't list kids shows any more. But, one of those precious custom
genres is still always kids and family related, no matter what we try. So,
we're giving up on fixing it, and just going to have a kid instead.

~~~
zheng
I almost spit out my coffee, very well done. I have the same issue in that I
set up my preferences, and told my wife (who uses it alot more but doesn't
look at the recommendations much, she generally already knows what she wants
to watch) to just not rate anything, so that it wouldn't be confused by my
ratings. All it took was one accidental 5* on Desperate Housewives and my
recommendations were shot.

It seems to me that the rating algorithm is a bit sensitive. Also, I wonder if
Netflix has considered giving people they option of multiple "personalities"
for the purpose of suggestions, queues, etc. I bet if they offered this for
something like $3/mo more, people would pay for it.

~~~
underwater
The DVD service allows multiple profiles for the same account; users can
switch between them to get their own queue and recommendations.

It looks like they explicitly remove that for streaming customers. I have no
idea why.

~~~
ronaldj
Maybe hoping people would just get one streaming account per person?

------
mgkimsal
It seems to me they'd be able to get a lot more metadata for their
recommendation engine if they worked on allowing multiple personas per
household account. Analyzing the viewing behaviour of people within the same
household would give them a treasure trove of more data to make better
recommendations for other people in similar households.

"You watched and liked XYZ, your 38 year old wife hates ABC, and you've got 3
daughters 8/10/12? They'll probably like these shows ...."

It seems they missed out on years of awesome data by not allowing for
segmenting by persona from the early days.

~~~
underwater
Posted below: "dailyllama [dead]: They actually have some support for this (or
at least they used to, and grandfathered some of us in)."

Their FAQ mentions it. But when I asked their support line was told it's only
available to subscribers of the DVD service.

~~~
endianswap
A year or so ago we discovered this at work and copied the URL for the setup
for this from someone with DVDs to someone who didn't, and they were able to
set it up for instant streaming, even though they didn't have the UI
presenting the feature visible. I'm not sure if it still works, but it was an
interesting bug to find, nevertheless.

------
flipside
_"Another important element in Netflix’ personalization is_ awareness _. We
want members to be aware of how we are adapting to their tastes. This not only
promotes trust in the system, but encourages members to give feedback that
will result in better recommendations. A different way of promoting trust with
the personalization component is to provide_ explanations _as to why we decide
to recommend a given movie or show. We are not recommending it because it
suits our business needs, but because it matches the information we have from
you: your explicit taste preferences and ratings, your viewing history, or
even your friends’ recommendations."_

Well, at least they're trying to demonstrate awareness and explain their
recommendations, even if they come up short.

Awareness? Am I missing something here because most people I've talked to
(several hundred for the record) feel like Netflix recommendations are hit or
miss, a black box that plateaus after a while. If anyone can explain to me how
Netflix conveys their awareness I'd love to hear it.

Speaking of explanations, sure, showing that I'm being recommended a movie
because of some other movies is sometimes helpful, but the rest of the time it
just reveals how poorly they understand me. It's can be pretty obvious that
the algorithms have no clue why I actually like certain movies, after all, how
could they? There are tons of things to like/dislike about any given movie,
but when that gets boiled down to 5-stars all that context gets lost.
Diminishing returns and noise will keep them from really understanding my
tastes with their current system.

Why am I ranting? Because I want to be understood but Netflix and other
personalization services still feel so damn impersonal. It's frustrating.

Anyways, we're working on a solution, sign up for our alpha at
<http://tagbax.com> or comment/vote to tell me how right/wrong I am. Thanks
for reading.

------
WiseWeasel
I'm actually mocking up a recommendation engine at the moment, and this
certainly gives me a lot to think about.

For example, I should probably be planning for more future dimensions
(ratings, metadata, etc.) so I can manipulate their interactions in a much
more elegant way as we add new data, rather than trying to manually flatten
everything out (combining metadata and ratings into a score for each item,
then calculating compatibility) for every possible combination, which seems
like a much less computationally expensive approach. I probably just need to
always keep every scrap of data we get, then find the right ways to combine
the data into a score with one routine, and compare the scores to calculate
compatibility with another. The hard part is figuring out how the score should
be calculated (I'll get to that in the morning).

Seems like this rabbit hole goes pretty deep...

------
mark_l_watson
I wish that they used a 10 star metric. Often I'll want to give a 3.5 or 4.5
rating - not possible. They need to use it a finer scale so customers get the
impression, at least, of finer control of how they rate movies.

~~~
StuffMaster
I wish there was at least one more - "OK". As in, I didn't "like" it, but I
don't regret watching it.

------
pagetruif
For people interested in the algorithms that won the Netflix Prize (including
the Matrix Factorization and Restricted Boltzmann Machine approaches mentioned
in the first section), there's a summary here:
[http://blog.echen.me/2011/10/24/winning-the-netflix-
prize-a-...](http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-
summary/)

~~~
mirsadm
I often wondered if this amount of effort is truly required and if it actually
provides results people can relate to. I used Netflix for a while and I find
the recommendations to be pretty average for me. Quite possibly it is because
I don't tend to rate most movies I watch. I wonder how often that is the case
for others though.

More importantly I just don't think you can predict what I want to watch based
on factors such as history, ratings, watch times etc. External factors such as
mood, recommendation from friend (not on Netflix) and curiosity almost always
play a large factor in choose the thing I watch next. These things are not
known to Netflix.

As a semi-educated guess I would say that simply providing a listing of
popular movies based on ratings/watch times per genre as well as similarity
would be enough for most people. It is easier to understand and how many
people want to watch something just because it is popular anyway?

~~~
speedracr
To me, the Netflix price is as much about marketing as it is about the quality
of the algorithm: Netflix was ahead of the pack (i.e. Blockbuster) even
before, but after spreading the word that it takes rocket scientists to even
slightly improve the existing algorithm, who would even think twice about
testing a DVD/ streaming provider other than Netflix?

------
sigil
Great read! Isn't this a bummer though? -- "On the topic of friends, we
recently released our Facebook connect feature in 46 out of the 47 countries
we operate – all but the US because of concerns with the VPPA law."

(I wonder to what extent it also constrains recommendations based on user
similarity.)

~~~
stephengillie
What's this VPPA?

<http://en.wikipedia.org/wiki/Video_Privacy_Protection_Act>

 _The Video Privacy Protection Act (VPPA) was a bill passed by the United
States Congress in 1988. Congress passed the VPPA after Robert Bork's video
rental history was published during his Supreme Court nomination. It makes any
"video tape service provider" that discloses rental information outside the
ordinary course of business liable for up to $2500 in actual damages._

(Covers DVD rentals too, and apparently paid-for streaming content.)

Wow, never heard of that law before...

~~~
waterlesscloud
It sucks. Prevents a lot of interesting possibilities. I see that it was
written with good intentions, but it's a great example of unintentional
consequences when it comes to legislation.

~~~
sbov
I disagree. Any company with my information should be liable for damages if
they disclose my information outside of normal business. Not just video
rentals.

~~~
ThomPete
But it's not Facebook that is publishing your history it would be yourself.

------
ck2
Instantwatcher and streamingsoon are far better than any netflix
recommendations in my experience.

------
damon_c
If you want to know more about the technical details...

[http://www.machinelearning.org/proceedings/icml2007/papers/4...](http://www.machinelearning.org/proceedings/icml2007/papers/407.pdf)

------
rmord
I remember seeing Xavier's talk at Strata 2012 (Santa Clara) on this, but not
sure if it is up on the web.

------
vph
recommendation is a good concept if you have things to recommend. Netflix
current catalog is seriously lacking. We were in the middle of watching the TV
show Spartacus when it was taken off (due to licensing problems).

------
Metapony
Screw netflix -- they stopped trying to even upsell you to dvd rentals if you
have instant. So if you search for something, and you have instant, they no
longer tell you that you _could_ rent the dvd. It's just plain dumb, but then
so was their last 'de-design'.

Also the instant que is not in the order we added the items in, and seems to
be arbitrary. It's just a terrible service with stale content. We'll be
cancelling soon.

~~~
DiabloD3
The reason you are getting downvoted is because you phrased your comment
badly.

HOWEVER, I do agree with you. I habitually do not use the web UI and only use
Netflix through my two Rokus. The web UI will tell you its on DVD, my Roku
won't.

Problem is, my Roku used to, so clearly the functionality exists, they just
shut it off. I'd like it to be put back, honestly.

