> How do we convince you that a title is worth watching?
That question seems revealing to me. Convincing me that a title is worth watching is good for business, I'm sure, but I want to decide what (and whether) to watch, not be convinced. Most titles are not worth watching.
I don't enjoy how relative and personal the recommendation system is right now, and I don't want more of it. The current recommendation system is showing me hundreds of titles that all say something like "97% match", plus or minus one percent, even though the majority of them are objectively crappy. It seems like the personalized ratings are all super inflated.
I want to know what other people thought and make my own decision, not be subject to a data collection and personalization system that filters everything through the eyes of a neural network that only knows what I've rated in the past.
It also feels like it's getting harder to branch out to genres I don't usually watch. The recommendations are giving me a metric ton of stuff that is similar (same genres, same actors) to whatever I rated highly, but not much that is dissimilar. It feels like I have to work harder now to find good stuff that is new or unusual.
Not to mention the surveillance chill of "how will watching this flick affect what the recommendation algorithm thinks of me." At least let me turn that shit off.
Maybe that's just context though, on say youtube where you have a billion vids and I'm just watching short music clips or programming vids, sure it may be helpful to show me stuff related to what I just watched. On Netflix just give me a directory and make it easy for me to browse, maybe show me the new releases etc. There's mostly no such things as 'similar' when it comes to feature length movies except for sequels and remakes.
Or I dunno, maybe they really just didn't have a single thing I wanted to watch, which is entirely possible.
>On Netflix just give me a directory and make it easy for me to browse
If they did that, you'd see just how small their catalog really is. For Netflix, the recommendation engine is vital camouflage for the fact that they just don't have very much content. Netflix have about 4,000 movies in their catalog, of which a substantial proportion are low-budget filler. Once you remove the obvious junk, you're left with a selection of movies that's no better than a Blockbuster store circa 2005.
Fundamentally, you don't get Netflix for movies. Maybe that's why you sign up originally but if you don't watch any TV shows, Netflix is probably not worth it for you. One of their senior people told me a few years ago that "People come for movies and they stay for TV." And that's probably still the case. I doubt its catalog is anywhere near the quality of a 2005 Blockbuster for films. Unfortunately, their DVD back catalog is also rotting away to the point where I'll probably end up dropping that subscription at some point.
> how will watching this flick affect what the recommendation algorithm thinks of me
Not to mention the fact that my wife and I use a single Netflix account. She watches things that I think are objectively garbage. I watch things she feels the same about. There are several things we enjoy watching together.
Just show me what you've added in the last month and make it easy for me to browse categories. That's all I want.
And how long is Kevin Hart's 3-year old stand up special going to be in the "recently added" section? It's been there for me for at least 8-9 months.
Just do what everyone does and create a guilty pleasures profile? Bonus, your guilty pleasures account will get better guilty pleasure recommendations.
I'm genuinely curious why? When I sit down to watch a movie I'm usually pretty sure if I want something with explosions or something with jokes or something with feels, is that atypical? Do you want pitch perfect and the green mile in the same set of recommendations?
I really just don't want any recommendations. They suck. Their algorithm has no idea what I want to watch. Maybe a list of things highly rated by other human beings that I could filter by genre would work. And not just other humans who've watched the same things I have.
But as the the immediate point, guess I'm just paranoid. I want to watch movies without data nerds peering through my windows, is that so wrong? I would very much like to be excluded from their preferences dataset entirely, but as an alternative I'll settle for just not watching stuff.
> How do we convince you that a title is worth watching?
Here's another quote that balances that one a bit:
> We also carefully determine the label for each observation by looking at the quality of engagement to avoid learning a model that recommends “clickbait” images: ones that entice a member to start playing but ultimately result in low-quality engagement.
I read that as saying that they want to avoid tricking you into watching stuff that you won't like. Certainly an interesting problem.
I agree with your points about how hard it feels to find new, interesting stuff on Netflix sometimes. The browsing interface feels like it's set up to make low effort discovery easy, but not to find content that's really great and unusual.
I often end up using a third-party site like Rotten Tomatoes to help me find something interesting. Perhaps that's because I trust independent reviews more than Netflix recommendations, especially now that Netflix features their own content so prominently.
It's like Netflix has (at least) two different population of users - those like us, who would like a catalogue of legally streamable good movies to choose from, and the other population for whom Netflix basically replaces TV, i.e. people who don't really care what exactly they watch, and thus are open to recommendations.
I suppose it's fair if they want to target that other population. Maybe they're majority. For me, their recommendation is only an utter annoyance, and it lost any value the moment they switched from objective ratings to "percent match".
Bingo. Everything you mention could be said for social media, news, entertainment, shopping, politics, fashion, etc.
Whether the "learning" is being done by neural networks built in software or human wetware makes no difference. We're optimizing for the pervasiveness and uniformity of echo chambers in nearly every corner of our lives. Those echo chambers are largely responsible for greater polarization of political views, lack of ideological diversity, the snowflake generation, and a host of other problems that seem to be getting worse by the day.
Sales and marketing. I blame this area of human endeavour for pretty much half of the shit that goes wrong on this planet.
It's because, instead of informing about their offer honestly and letting people decide, people figured out they have to convince buyers. Given that this is mostly a highly-competitive zero-sum game (for a given established market segment you're in, you're competing with others for limited amount of potential customers), I suppose me complaining about this is like complaining that the water is wet, or that speed of light is too slow.
But pause for a minute and consider, how so many of this bullshit - including "greater polarization of political views, lack of ideological diversity, the snowflake generation, and a host of other problems that seem to be getting worse by the day" - all boil down to individual entrepreneurs who engage in convincing to bump their income a tiny little bit each step of the way.
Precisely. This is a strictly business focused/profit gaining feature, not a consumer feature. The consumer's actual wants and desires seem to have been entirely left out. The question "what does the customer want" is shaped for the assumption "The customer wants what will net our company more profit"
Their whole goal is to give you a service that keeps you paying every month and encouraging you to tell your acquaintances that they should sign up too.
So I think your interest and their interests are aligned?
’Interests’ are more complex and layered than that. Am I interested in going to the gym or is the underlying interest to feel healthy and fit? A brain may be easily misled to believe an expensive colorful drink would satisfy the interest to feel healthy. Maybe the brain kinda knows it should do x to really be healthy but y is easier. It might advertise to its brainy friends that y ‘ is pretty good in fact!’ to legitimize its preferences.
It’s your own responsibility to decide what to want and what not to want, but these days it’s pretty hard to fight the hundreds and hundreds of brightest minds that work full-time to convince you to like something. They know much more about the inner workings of your brain than you do, and you might not be equipped to tip the odds of manipulation in my favor.
I recently deleted Facebook and it was a real mind blower to realize how much they impacted my brain chemistry even though I didn’t feel like I used their service much. Since then I’m even less trusting of corporations armed with big data, big brains, big money and supposedly aligned incentives.
Years ago when there still was a public API I wrote a super simple script that just listed movies in order by how much Netflix thought I'd like them. This was the best Netflix interface I've used so far. While the predictions weren't super accurate they were accurate enough to quickly scan over it and find interesting stuff. They also we accurate enough to easily notice at what point in the list the was no reason to look further down. If I had spend some more time to substraction what I've already watched and maybe also make it also sortable by IMDb ratings it would have been perfect. Also no fing photos. Just give me the title and ratings, maybe just maybe the release year. At least on the overview.
I believe Netflix moved to the current model because they don't care about you spending your time well and in addition need to fudge their catalog. It's ultimately wasting user's time. Before they had the new match percentage it regularly would suggest movies that they themselves predicted I'd hate. WTF?!
At this point my favorite movie platform is mubi. Just 30 well selected movies. I can quickly tell what's new to their catalog and also if I should just close the app and do something else.
I can only assume that a significant goal of the Netflix interface is hiding what they don't have and making it less obvious when things go away.
Even basic parts of the UI like "My List" or "Continue watching" aren't reliably available in the same places.
It is somewhat funny to see, for many years now, recommendation and ratings take such a heavy back seat when they were previously one of Netflix's most famous features.
That is something I still don't understand about the modern Netflix recommendation engine. Before the percentage system, it would literally recommend things to me that it predicted I'd give two stars. It's probably still recommending things in that range today, but I gave up consulting it.
HBO Go's interface has its issues, but it makes it relatively easy to see what's new this month or so. Well, I can tell usually it's not enough to justify staying subscribed. So I subscribe ~4-6 months a year watch what I want to see and then unsubscribe. I'm pretty certain I'd follow a similar pattern with Netflix if there isn't wasn't obfuscating their catalog.
All due respect for the authors and engineers that worked on this, but this solves a problem that does not exist. In fact, it has produced confusion as my wife and I can't find a title if the artwork has changed or is different on mobile vs web vs FireTV, etc and the sands are shifting on each depending on whose account we are on.
You're gonna get higher click through in the short term but lower satisfaction in the longer term. This "up-and-to-the-right" disease will erode your credibility by not treating artwork as canon in subtle ways and alienate your audience who are disappointed Good Will Hunting, in this example is more drama/romance than comedy, and that Robin Williams, (usually) a comedian, is the most serious character (he graduated from Juliard).
tl;dr Long time personalization product manager here, don't do this: it will hurt you
My thoughts exactly as how much confusion this causes. It has happened to me several times that I'm scanning the catalog for a movie that my graphic memory had previously located, just to end up frustrated at first for not being able to find it and angry after realizing Netflix simply decides to A/B test the artwork.
This is like having a vinyl music collection made up of hundreds of items where the covers change randomly. Please Netflix, stop doing this.
Even for films you haven't seen, it's a pretty terrible system. I didn't see Good Will Hunting until maybe a decade after it came out, but I was aware of it and knew it starred Matt Damon. If I had looked for it on Netflix and found a cover with Robin Williams, I might have wondered - was this another film with the same title?
> Aren't they proposing this would be shown to you for titles you haven't seen, and haven't made an association with yet?
Haven't seen on Netflix, using your (sub)account. This breaks in an annoying way when you saw the artwork previously at your friend's party, or on your spouse's subaccount, etc., and are now skimming the catalogue for the movie (maybe you don't remember the exact title, but know it's in a particular genre list).
They don’t use click through as their success metric.
It is based on their engagement score for the title. With this change, folks are consuming more content, so I don’t think it is correct saying this will hurt them based on your own negative reaction.
5 years ago I wrote the recommendation system that Netflix uses (and has degraded since then). One major problem is in the past certain senior Netflix managers are only interested in self promotion (I would hate to extrapolate to the current ones - even if the extrapolation is reasonable). A/B tests are a perfect device for this. What is a better recommender? They were not interested in improving the product. It is easier to win by politics/lying/obfuscation/omission/plagiarism then come up with better ideas. If the company goes down they move on with a good resume and the games they played (for instance USPTO fraud), are hidden.
Some companies are like this. For instance the "Netflix prize team" at Verizon/Yahoo refuse to share recommendations data with other teams leaving them nothing to do. They work in a bubble and will actively try and remove anyone who might be a competitor.
It's sad that Netflix decided to pursue this work (+ other even more brain-dead projects like rewriting the command-line parser, "switching to OO" -
examples of Xavier's initiatives). They could of 5 years ago pushed the beta system that has 40% extra performance. (I'm currently at a factor of 3 better performance).
This was my thought after seeing the Pulp Fiction example. The Uma image is iconic, and a lot of work has been done to get it littered around big box stores and plastered on dorm room walls. Using a different image ignores the strength of that brand.
I think this highlights an inherent tension between marketing and art.
When I create such and such a piece as an artist I want to present it a certain way, even down to the marketing material--I want it all to express the correct vision--i.e. there is one 'correct' promotional image for it.
Marketers, contrarily strictly have the utilitarian principle of conversion in mind--they want to manipulate my vision to make it a better fit for as wide an audience as possible and in so doing diminish and cheapen the art to target certain subsets or groups who may not actually fit the intended audience when you consider the work as a whole. Art is reduced to a profit making instrument.
That's precisely what this method attempts--cast a wider net by deceiving the fish and ignoring the wishes of the fisherman, who, only wanting trout, now gets the whole biosphere of the sea in his boat.
Its not very honest, and will probably make people upset unless the content based recommendation algorithm aligns perfectly with the image focused one (i.e. the image is only manipulated to fit your taste after it's ascertained the content as a whole and on its own fits your taste, and not only a subset of that content, i.e. one or two brief romantic scenes in a movie that has as its actual subject matter something entirely unromantic).
Author of any kind is not an abstract "artist" from an ivory tower that receives his work from a muse in a fit of inspiration - it's just an image that helps to sell copies. He's a marketer, too.
He has to sell his image and unique style. When he works, he has to imagine what effect will his work have on the audience - which is exactly a marketer's skill.
Also, these "versions" of product are quite common in different forms of art. Songs have radio versions, album versions, remixes and simply alternate releases. Whole albums have different masters (released for different medium and audience). Movies have director's cuts, movie cuts, TV adaptations (where they sometime get serialized). Books composed from chapters printed periodically in magazines, with author responding to reader's feedback (which was much more common in 19th century, but still).
Historically the promo artwork has usually differed in different countries for example, often a lot. So I am not sure the "single vision" is really the case.
All this and they still haven't done anything about the basic problem with the front page: I don't want to see things that I've already watched or that I'm explicitly not interested in.
I'd just like to be able to do things in less than like 20 clicks + mouseovers + impromptu game of "dodge the UI" + now do all that at single-digit FPS.
What the heck happened to their UI? It used to be quick, and reasonably useful. At this point, I'd rather go to Amazon and try to guess which search result (and on which page!) is the video I've already purchased.
I think they will never implement it. If all the shit and stuff you've already watched was filtered out and every movie was only listed ones you'd likely see a fairly empty page.
I think there's a large enough repository that there would always be something to show. Why wouldn't Netflix prefer to show me things I might want to watch (even if it's a low probability) over things I said I definitely don't want to watch.
For me, I think it basically just reduces my engagement. If my recommendations are filled with things I've already decided I don't want to see, I'll just go do something else.
I want this so bad. Right now, Netflix thinks I want to watch animated christmas movies for kids. If that's what it comes up with after looking at my ratings and watch history, their algorithm is fundamentally broken.
It reminds me of Pandora. Even after multiple thumbs down, I get the same song played. I mean, why ask? It annoys me more than if they just played it rather then pretend they have given me control.
> This is yet another way Netflix differs from traditional media offerings: we don’t have one product but over a 100 million different products with one for each of our members with personalized recommendations and personalized visuals.
What? If I go over to a friend's house, I don't think of myself as using a different product. I don't want to use a different product. This advice seems to go against everything I've heard about brands and recognition.
One practical point anyone implementing bandits should be aware of is time correlation. Bandits are awesome if your data behaves the same going forward as it did before. But when you change your A/B split on Friday based on what you learned over the week, then suddenly whether someone saw A or B is correlated with being the kind of user that visits on weekends. That leads to selection bias, which is a pain in the ass and screws things up. It can be time of day, day of week, and even season (I bet all of our jobs will be a bit unusual going in to this coming week, for example). The problem's present on many time scales.
You can deal it, but everyone should be aware of this before doing the naive thing of just throwing bandits at things. Sometimes some regret is worth the plain validity of an RCT, and sometimes not.
This thing is hilarious. I’ve noticed that Netflix far prefers to display black faces to me, and women’s faces to my wife. Personally I think they should spend their time making sure I can read the title of the show, but whatever...
Netflix gets huge props for paying their engineers handsomely, especially the infrastructure guys. And they still spend more on content acquisition than they do on payroll. The fact that their lackluster catalog costs them more than a staff full of $400k/yr+ engineers and executives sheds light on the real problem: it's too goddamn expensive to get the licensing rights to show a movie or television show.
My wife and I own two gyms and payroll is our largest expense by multiple orders of magnitude. Every business I've ever looked at or been involved in, payroll has been the largest expense, often by one order of magnitude or more. The fact that licensing still beats out Netflix's payroll is insane.
Why should you care about any particular title we recommend? What can we say about a new and unfamiliar title that will pique your interest? How do we convince you that a title is worth watching?
How about, instead of overpersonalizing everything, you say "we recommend X because you liked Y and people like you also like X."
I wish sites were more up-front about it. In many places, I get a recommendation and I wonder, "why on Earth did the site just suggest that?".
Steam does a little bit of what you suggest for games. When I browse their recommendation queue for me and I see a weird title, I can look down and see a little text that say something like "This game is suggested because it's new on Steam", or "because you've played similar games", or "because you've played games tagged XYZ", etc. It makes me feel that they don't pull the queue out of their collective ass.
The whole "a machine decides what I watch"-thing scares me a bit. It creates a world where people only listen to the exact thing they like. Like a reddit/HN echo chamber, but instead of being designed for communities, it is tailored to one person. An individual echo chamber.
Of course it is nice to listen to music we love and to see movies that fit our past viewing experience, but what about putting some plain randomness in there too. Wikipedia has a "random article" button. The online content world is not random enough.
The "My List" is terrible, fix that. On XBox360 there is no gridded "My List", so I have to scroll through pages of a single tape. If I've watched all episodes of a series, have it filter to the very bottom until there is a new episode which I HAVEN'T yet watched, and only then add a red "new episodes" tag and filter it to the top. That's just the start of it. With all the ridiculous easy-fix problems, you have the audacity to post this crap? These kinds of show-off posts make me cringe at how poorly you, Netflix, are being managed. You're showing off the wrong thing! It's like if I hire a kid to mow my lawn, and I come back 2hrs later and instead of the lawn being mowed he shows me with great pride how he planted a fucking pineapple in the fucking back corner of the yard. I just wanted my fucking lawn mowed!
I'm aware of a general screaming into the void complaining here, so I'll add two different comments.
About the article:
Very interesting idea, and the Replay is a fascinating and in hindsight obvious way of being able to quickly test out new approaches without waiting for the data collection.
Screaming into the void:
If you're able to do all this, why do you still write "NEW EPISODES" on the thumbnails of shows you know I've already watched? Why do I have broken images constantly in my android app? Why can I not turn off autoplay of trailers (meaning if I don't want to have things spoiled I need to keep flicking around)? Why do you suggest I watch things I've 1. Already seen and 2. Told you I hated?
So... as useful as Amazon's 'products we think you will like', an echo chamber of previous purchases ala Google's search bubble. This feels reminiscent of Facebook's attempts at emotional manipulation experiments.
Well, since you removed rating by stars and replaced it with nigh useless thumbs system, you don't seem to be very interested in personalising what I want to watch. And it shows.
Since the recommended titles are clearly very focused on Netflix production, I just ignore any recommendations altogether and just browse all new arrivals every week...
I hate shit like that. I pretty quickly memorise covers and then it's agony to find anything when I have to read every single freakin' title in their super small UI :/ Do. NOT. Do. THAT.
Amazon Prime shows me rating from IMDB (or Rotten Tomatoes), which I find more useful in making a decision on what to watch, than these tweaks. For me, these are cool, but will not move the needle much.
Also I am not sure if I want a different "Poster" for "My Cousin Vinny". Ever.
Can they also do some explaining on how they get to those seemingly completely irrelevant match percentage scores? I have to _avoid_ things that have too high of a match, to the point that I don't think it's just a crappy match system but rather one that actively tries to push content on me that I _hate_.
Now let's talk how Netflix "personalizes" Most Popular and Trending titles - on 3 different accounts I've seen there 3 different set of movies in each of the hit lists.
That question seems revealing to me. Convincing me that a title is worth watching is good for business, I'm sure, but I want to decide what (and whether) to watch, not be convinced. Most titles are not worth watching.
I don't enjoy how relative and personal the recommendation system is right now, and I don't want more of it. The current recommendation system is showing me hundreds of titles that all say something like "97% match", plus or minus one percent, even though the majority of them are objectively crappy. It seems like the personalized ratings are all super inflated.
I want to know what other people thought and make my own decision, not be subject to a data collection and personalization system that filters everything through the eyes of a neural network that only knows what I've rated in the past.
It also feels like it's getting harder to branch out to genres I don't usually watch. The recommendations are giving me a metric ton of stuff that is similar (same genres, same actors) to whatever I rated highly, but not much that is dissimilar. It feels like I have to work harder now to find good stuff that is new or unusual.