
Artwork Personalization at Netflix - dingdongding
https://medium.com/netflix-techblog/artwork-personalization-c589f074ad76
======
dahart
> How do we convince you that a title is worth watching?

That question seems revealing to me. Convincing me that a title is worth
watching is good for business, I'm sure, but I want to decide what (and
whether) to watch, not be convinced. Most titles are not worth watching.

I don't enjoy how relative and personal the recommendation system is right
now, and I don't want more of it. The current recommendation system is showing
me hundreds of titles that all say something like "97% match", plus or minus
one percent, even though the majority of them are objectively crappy. It seems
like the personalized ratings are all super inflated.

I want to know what other people thought and make my own decision, not be
subject to a data collection and personalization system that filters
everything through the eyes of a neural network that only knows what I've
rated in the past.

It also feels like it's getting harder to branch out to genres I don't usually
watch. The recommendations are giving me a metric ton of stuff that is similar
(same genres, same actors) to whatever I rated highly, but not much that is
dissimilar. It feels like I have to work harder now to find good stuff that is
new or unusual.

~~~
nol13
This, please, pretty please.

Not to mention the surveillance chill of "how will watching this flick affect
what the recommendation algorithm thinks of me." At least let me turn that
shit off.

Maybe that's just context though, on say youtube where you have a billion vids
and I'm just watching short music clips or programming vids, sure it may be
helpful to show me stuff related to what I just watched. On Netflix just give
me a directory and make it easy for me to browse, maybe show me the new
releases etc. There's mostly no such things as 'similar' when it comes to
feature length movies except for sequels and remakes.

Or I dunno, maybe they really just didn't have a single thing I wanted to
watch, which is entirely possible.

~~~
jdietrich
>On Netflix just give me a directory and make it easy for me to browse

If they did that, you'd see just how small their catalog really is. For
Netflix, the recommendation engine is vital camouflage for the fact that they
just don't have very much content. Netflix have about 4,000 movies in their
catalog, of which a substantial proportion are low-budget filler. Once you
remove the obvious junk, you're left with a selection of movies that's no
better than a Blockbuster store circa 2005.

~~~
ghaff
Fundamentally, you don't get Netflix for movies. Maybe that's why you sign up
originally but if you don't watch any TV shows, Netflix is probably not worth
it for you. One of their senior people told me a few years ago that "People
come for movies and they stay for TV." And that's probably still the case. I
doubt its catalog is anywhere near the quality of a 2005 Blockbuster for
films. Unfortunately, their DVD back catalog is also rotting away to the point
where I'll probably end up dropping that subscription at some point.

~~~
Fnoord
Agreed w/the rest of your comment but "[...] they stay for TV?" I take it they
and/or you meant "series".

------
ajmurmann
Years ago when there still was a public API I wrote a super simple script that
just listed movies in order by how much Netflix thought I'd like them. This
was the best Netflix interface I've used so far. While the predictions weren't
super accurate they were accurate enough to quickly scan over it and find
interesting stuff. They also we accurate enough to easily notice at what point
in the list the was no reason to look further down. If I had spend some more
time to substraction what I've already watched and maybe also make it also
sortable by IMDb ratings it would have been perfect. Also no fing photos. Just
give me the title and ratings, maybe just maybe the release year. At least on
the overview.

I believe Netflix moved to the current model because they don't care about you
spending your time well and in addition need to fudge their catalog. It's
ultimately wasting user's time. Before they had the new match percentage it
regularly would suggest movies that they themselves predicted I'd hate. WTF?!

At this point my favorite movie platform is mubi. Just 30 well selected
movies. I can quickly tell what's new to their catalog and also if I should
just close the app and do something else.

~~~
mortenjorck
That is something I still don't understand about the modern Netflix
recommendation engine. Before the percentage system, it would literally
recommend things to me that it predicted I'd give two stars. It's probably
still recommending things in that range today, but I gave up consulting it.

~~~
ajmurmann
HBO Go's interface has its issues, but it makes it relatively easy to see
what's new this month or so. Well, I can tell usually it's not enough to
justify staying subscribed. So I subscribe ~4-6 months a year watch what I
want to see and then unsubscribe. I'm pretty certain I'd follow a similar
pattern with Netflix if there isn't wasn't obfuscating their catalog.

------
timfrietas
All due respect for the authors and engineers that worked on this, but this
solves a problem that does not exist. In fact, it has produced confusion as my
wife and I can't find a title if the artwork has changed or is different on
mobile vs web vs FireTV, etc and the sands are shifting on each depending on
whose account we are on.

You're gonna get higher click through in the short term but lower satisfaction
in the longer term. This "up-and-to-the-right" disease will erode your
credibility by not treating artwork as canon in subtle ways and alienate your
audience who are disappointed Good Will Hunting, in this example is more
drama/romance than comedy, and that Robin Williams, (usually) a comedian, is
the most serious character (he graduated from Juliard).

tl;dr Long time personalization product manager here, don't do this: it will
hurt you

~~~
ec109685
They don’t use click through as their success metric.

It is based on their engagement score for the title. With this change, folks
are consuming more content, so I don’t think it is correct saying this will
hurt them based on your own negative reaction.

~~~
krzat
I think it's a mistake to optimize for most hours watched. The price for user
is constant, but cost for Netflix - probably not.

For example I watched some shitty movie recommended by Netflix and I feel less
satisfied with the service than if I didn't watch it at all.

~~~
user5994461
That's the best explanation I have seen so far. They're optimizing right but
maybe for the wrong metrics.

------
voidhorse
I think this highlights an inherent tension between marketing and art.

When I create such and such a piece as an artist I want to present it a
certain way, even down to the marketing material--I want it all to express the
correct vision--i.e. there is one 'correct' promotional image for it.

Marketers, contrarily strictly have the utilitarian principle of conversion in
mind--they want to manipulate my vision to make it a better fit for as _wide
an audience as possible_ and in so doing diminish and cheapen the art to
target certain subsets or groups who may not actually fit the intended
audience when you consider the work as a whole. Art is reduced to a profit
making instrument.

That's precisely what this method attempts--cast a wider net by deceiving the
fish and ignoring the wishes of the fisherman, who, only wanting trout, now
gets the whole biosphere of the sea in his boat.

Its not very honest, and will probably make people upset unless the content
based recommendation algorithm aligns perfectly with the image focused one
(i.e. the image is only manipulated to fit your taste after it's ascertained
the content as a whole and on its own fits your taste, and not only a subset
of that content, i.e. one or two brief romantic scenes in a movie that has as
its actual subject matter something entirely unromantic).

~~~
golergka
Author of any kind is not an abstract "artist" from an ivory tower that
receives his work from a muse in a fit of inspiration - it's just an image
that helps to sell copies. He's a marketer, too. He has to sell his image and
unique style. When he works, he has to imagine what effect will his work have
on the audience - which is exactly a marketer's skill.

Also, these "versions" of product are quite common in different forms of art.
Songs have radio versions, album versions, remixes and simply alternate
releases. Whole albums have different masters (released for different medium
and audience). Movies have director's cuts, movie cuts, TV adaptations (where
they sometime get serialized). Books composed from chapters printed
periodically in magazines, with author responding to reader's feedback (which
was much more common in 19th century, but still).

------
crooked-v
All this and they still haven't done anything about the basic problem with the
front page: I don't want to see things that I've already watched or that I'm
explicitly not interested in.

~~~
michaelmior
I do really wish I could tell Netflix about things I never want to watch.

~~~
ajmurmann
I think they will never implement it. If all the shit and stuff you've already
watched was filtered out and every movie was only listed ones you'd likely see
a fairly empty page.

~~~
michaelmior
I think there's a large enough repository that there would always be something
to show. Why wouldn't Netflix prefer to show me things I _might_ want to watch
(even if it's a low probability) over things I said I definitely _don 't_ want
to watch.

For me, I think it basically just reduces my engagement. If my recommendations
are filled with things I've already decided I don't want to see, I'll just go
do something else.

------
Jasper_
> This is yet another way Netflix differs from traditional media offerings: we
> don’t have one product but over a 100 million different products with one
> for each of our members with personalized recommendations and personalized
> visuals.

What? If I go over to a friend's house, I don't think of myself as using a
different product. I don't _want_ to use a different product. This advice
seems to go against everything I've heard about brands and recognition.

~~~
TeMPOraL
Indeed.

My perception is, they're a big enough brand now that they don't have to care
anymore.

------
imh
One practical point anyone implementing bandits should be aware of is time
correlation. Bandits are awesome if your data behaves the same going forward
as it did before. But when you change your A/B split on Friday based on what
you learned over the week, then suddenly whether someone saw A or B is
correlated with being the kind of user that visits on weekends. That leads to
selection bias, which is a pain in the ass and screws things up. It can be
time of day, day of week, and even season (I bet all of our jobs will be a bit
unusual going in to this coming week, for example). The problem's present on
many time scales.

You can deal it, but everyone should be aware of this before doing the naive
thing of just throwing bandits at things. Sometimes some regret is worth the
plain validity of an RCT, and sometimes not.

------
moomin
This thing is hilarious. I’ve noticed that Netflix far prefers to display
black faces to me, and women’s faces to my wife. Personally I think they
should spend their time making sure I can read the title of the show, but
whatever...

------
thom
Just fix /browse/just-added, the only useful page on all of netflix.com. None
of this algorithmic crap matters, the catalogue is _tiny_.

------
pc86
Netflix gets huge props for paying their engineers handsomely, especially the
infrastructure guys. And they still spend more on content acquisition than
they do on payroll. The fact that their lackluster catalog costs them more
than a staff full of $400k/yr+ engineers and executives sheds light on the
real problem: it's too goddamn expensive to get the licensing rights to show a
movie or television show.

My wife and I own two gyms and payroll is our largest expense by _multiple
orders of magnitude_. Every business I've ever looked at or been involved in,
payroll has been the largest expense, often by one order of magnitude or more.
The fact that _licensing_ still beats out Netflix's payroll is insane.

~~~
exhilaration
While your observation is interesting, it seems a bit silly to compare the
operating expenses of a global media distributor to a local gym.

------
nerdponx
_Why should you care about any particular title we recommend? What can we say
about a new and unfamiliar title that will pique your interest? How do we
convince you that a title is worth watching?_

How about, instead of overpersonalizing everything, you say "we recommend X
because you liked Y and people like you also like X."

~~~
TeMPOraL
OH GOD YES.

I wish sites were more up-front about it. In many places, I get a
recommendation and I wonder, "why on Earth did the site just suggest that?".

Steam does a little bit of what you suggest for games. When I browse their
recommendation queue for me and I see a weird title, I can look down and see a
little text that say something like "This game is suggested because it's new
on Steam", or "because you've played similar games", or "because you've played
games tagged XYZ", etc. It makes me feel that they don't pull the queue out of
their collective ass.

------
antoineMoPa
The whole "a machine decides what I watch"-thing scares me a bit. It creates a
world where people only listen to the exact thing they like. Like a reddit/HN
echo chamber, but instead of being designed for communities, it is tailored to
one person. An individual echo chamber.

Of course it is nice to listen to music we love and to see movies that fit our
past viewing experience, but what about putting some plain randomness in there
too. Wikipedia has a "random article" button. The online content world is not
random enough.

------
jmpeax
The "My List" is terrible, fix that. On XBox360 there is no gridded "My List",
so I have to scroll through pages of a single tape. If I've watched all
episodes of a series, have it filter to the very bottom until there is a new
episode which I HAVEN'T yet watched, and only then add a red "new episodes"
tag and filter it to the top. That's just the start of it. With all the
ridiculous easy-fix problems, you have the audacity to post this crap? These
kinds of show-off posts make me cringe at how poorly you, Netflix, are being
managed. You're showing off the wrong thing! It's like if I hire a kid to mow
my lawn, and I come back 2hrs later and instead of the lawn being mowed he
shows me with great pride how he planted a fucking pineapple in the fucking
back corner of the yard. I just wanted my fucking lawn mowed!

------
IanCal
I'm aware of a general screaming into the void complaining here, so I'll add
two different comments.

About the article:

Very interesting idea, and the Replay is a fascinating and in hindsight
obvious way of being able to quickly test out new approaches without waiting
for the data collection.

Screaming into the void:

If you're able to do all this, why do you still write "NEW EPISODES" on the
thumbnails of shows _you know I 've already watched_? Why do I have broken
images _constantly_ in my android app? Why can I not turn off autoplay of
trailers (meaning if I don't want to have things spoiled I need to keep
flicking around)? Why do you suggest I watch things I've 1. Already seen and
2. Told you I hated?

------
have_faith
So... as useful as Amazon's 'products we think you will like', an echo chamber
of previous purchases ala Google's search bubble. This feels reminiscent of
Facebook's attempts at emotional manipulation experiments.

------
Aoyagi
Well, since you removed rating by stars and replaced it with nigh useless
thumbs system, you don't seem to be very interested in personalising what I
want to watch. And it shows.

Since the recommended titles are clearly very focused on Netflix production, I
just ignore any recommendations altogether and just browse all new arrivals
every week...

~~~
runj__
>I just ignore any recommendations altogether and just browse all new arrivals
every week...

I do the same only to find that there's nothing interesting there and watch
Arrested Development again.

------
pawelkomarnicki
I hate shit like that. I pretty quickly memorise covers and then it's agony to
find anything when I have to read every single freakin' title in their super
small UI :/ Do. NOT. Do. THAT.

------
somberi
Amazon Prime shows me rating from IMDB (or Rotten Tomatoes), which I find more
useful in making a decision on what to watch, than these tweaks. For me, these
are cool, but will not move the needle much.

Also I am not sure if I want a different "Poster" for "My Cousin Vinny". Ever.

------
temp
Can they also do some explaining on how they get to those seemingly completely
irrelevant match percentage scores? I have to _avoid_ things that have too
high of a match, to the point that I don't think it's just a crappy match
system but rather one that actively tries to push content on me that I _hate_.

------
uladzislau
Now let's talk how Netflix "personalizes" Most Popular and Trending titles -
on 3 different accounts I've seen there 3 different set of movies in each of
the hit lists.

