Hacker News new | past | comments | ask | show | jobs | submit login
Netflix: Lessons in Experimentation (aakashg.com)
85 points by gradientgarden on Feb 1, 2022 | hide | past | favorite | 35 comments



If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List?

Several series I’ve enjoyed watching several years ago now appear on Netflix. Too late for me on Netflix, but as I liked them I rate them with thumbs up, hoping that Netflix would give me better recommendations. Instead of recommending other movies or series, to my dismay Netflix starts recommending the series I’ve already watched elsewhere. As I can only rate a series I’ve watched, why on earth does Netflix recommend something I’ve already watched?

The diversification of Netflix is great, but most of it is too little or too late for my taste. Still better than Disney or Apple TV+ where I could find 1 or 2 shows of my liking. Looking forward to a couple of months of HBO Max once it is available in my region. HBO Europe made some award winning documentaries and I’m hoping to find more that I haven’t watched.

Luckily for me there is an alternative for the streaming platform as for the past several decades every week my local TV stations have 1 or 2 great movies and maybe a new serie, selected not by experiments but by real people of flesh and blod.


They used to have good recommendations for their DVD service with star ratings. The rental catalog was huge and varied then so a lot of interest areas were well covered and easy to extrapolate from your ratings and your common cohort. I suspect the problem they have now is most of the top tier content is gone and it's 90% filler that isn't compelling to watch.


I think the recommendations are not good on Netflix because the content is mostly not good. Lots of Netflix Original stuff that lacks staying power and seems designed to pander and not to actually be good. Fortunately, as time marches on, Turner Classic Movies is starting to show films from the 70s and 80s (I guess something can only be a "classic" if it's at least 40 years old), so I imagine that TCM will get a better and better selection in the coming years. If you have a cable subscription, you can stream their rotating selection of films online.

I also find that Hulu generally has better options for TV series than Netflix. You have to deal with ads unless you pay extra for the premium subscription, but it's not like they aren't harvesting your data anyway, and it's easy to just mute the ads (I always muted them on cable, too) and go do something else. It's also still a lot fewer ads than on cable TV.


I can only assume that the majority of people like a) rewatching things they’ve already seen, b) like watching things they said they don’t like, and c) watch things from categories they don’t like.

Because that’s what Netflix has been doing to me pretty much since they became a streaming service.


> If their experiment are zo good, why are the recommendations so bad that I had to turn off the feature that adds items to My List?

They've gotten very good at grabbing attention with cover art, but then you click on it and quickly realise its crap.

In other words, they've put appealing covers on all the boxes, so now you pick ones that are crap only to abandon them 20 minutes later.

I wonder if abandonment rate is a metric too.


> I wonder if abandonment rate is a metric too.

Apparently they categorize the type of viewer you are based on when you abandon the content. [1] mentions starter, watcher, and completer as viewer categories based on how much of the content they view.

[1] https://www.fool.com/investing/2019/12/06/how-netflix-decide...


Never realizing that "abandoner" is likely someone whose suggestions and bad in the first place.


> why are the recommendations so bad

Without knowing anything about how it works internally I'd guess that the recommendations are not simply about what's best for the user but also what's best for Netflix (pushing Netflix originals).


If you want something selected and curated by actual humans, I recommend checking out Mubi. Lots of odd ball, but phenomenal movies on there. They started out having thirty movies on there and replacing the oldest every day with a new movie. It's often times a few in a row by a specific director or around a theme. They now also a have decent library in addition.

Almost every movie I find truly memorable in they just few years came to me via Mubi.


Recommendations aren't really chosen by "experiments". And certainly, they aren't A/B testing budgeting and development of shows.

Recommendation engines can be improved with experiments. And they - along with budgeting and show development (at least at Netflix) - are "driven by big data".

Further, it doesn't matter if you - personally - like a feature. If that feature is moving the numbers and passes the A/B test, it will make itself into the app.

They run thousands of experiments. Of course there are going to be several that you - personally - do not like.


> Netflix abandoned the ‘large catalog’ strategy for a ‘right for you, personalized catalog’ strategy.

The "right for [me]" catalog seems to contain a lot of "get the viewer hooked on the first two seasons of a show; don't catalog the latter seasons".


I find the recommendations rarely useful. But what really bugs me is that the same series / movie typically shows up in all kinds of lists. Over and over and over. Besides the categories - often filled with stuff not matching the label - I see the same stuff repeatedly in "trending now", "top 10 today", "only on Netflix", "popular on Netflix".


Netflix experimentation seems to have suffered in the past year IMHO, with worsening recommendations and no simple way for users to provide feedback. And bugs are creeping in-- this week I've hit two different bugs in the Netflix recommendation engine, so something's amiss.

The experiment I would love to see is an simple icon that means "I don't want to watch this".


It's been pretty apparent for me that their who focus changed a couple of years ago from "here's some stuff you might enjoy watching" to "here's the stuff we want you to watch" or much like Google ads, "here's something you've already watched, watch it again"

I'm almost at the point of cancelling my Netflix subscription simply because it's so much effort to find something of interest now.


For the love of God, can we please get an "Only show me things in my native language" option?


There already exists a simple icon for "I don't want to watch this": the thumbs down button. Clicking it is supposed to remove the show from your recommendations (after a day or so, to clear the caches).


It turns out that Netflix thumbs down conflicts with thumbs up.

For example, pick a movie that you like, yet don't ever want to watch on Netflix; perhaps you've already seen the movie on a different service. Do you click thumbs up ("I like this") or thumbs down ("Not for me")?

Next, if you watch some of the movie, then it shows up in your "Currently Watching" list, with icons for thumbs up ("I like this but don't want to continue watching"), thumbs down ("I don't like this"), and a broom ("Just cleaning up").

What I am suggesting as an experiment is a choice that means "I like this but don't want it on my main page".


We’re hiring to do more of what’s in the article: https://jobs.netflix.com/search?q=Experimentation


Do create a button called "wipe my recommendations and start anew"


There already exists button for wiping your recommendations and starting anew: It's labeled 'add profile'. :)

(Otherwise, you can scroll through you watch history clicking delete a zillion times.)


I assume you probably mean well and are trying to help the GP, but this comes off as a totally tone-deaf PR response.

The GP doesn't want to create a new profile, but they still want to avoid "clicking delete a zillion times" in their watch history.

In my opinion you should acknowledge that the user has an actual problem that your team could solve by providing them that button THEN offer the workaround of creating a new profile with apologies instead of a potentially snarky smile.


I don't work at Netflix. I'm not Netflix PR. The smile was meant in good spirit to acknowledge the hackiness of the solution. I don't see why this solution doesn't solve the problem. Creating a new profile is effectively the same as resetting your profile. It absolutely is a solution.

Same thing with Facebook or any other service. If you want to "reset" your Facebook profile, just create a new one.


Maybe they mixed up your username with mine? And if this is the Ted Sanders I’m thinking of, good place to mention that you did previously work at Netflix. Offering help with a hacky solution and smiley face would be consistent with the Ted I met.


I was hoping that a benefit of leaving Netflix is that I could finally start offering tips & tricks to people in public discussion threads... but perhaps that was a bad idea after all.

Hi Travis :)


Neither are solutions.

In general, any recommendation system should come with two things:

- an option to clear all recommendations and start from scratch

- opt out of recommendations entirely


Yes, someone with some basic recommendation engine knowledge please apply! A basic rip-off website streaming site has a better recommendation engine AND ux then this company. Was given a subscruption as a present and it dumped it in the bin a few weeks later after. It seems like the algorithm is a basic highlight whatever scam show is pumped by the bingers at the time. Not even the show i last watched (consitantly everyday) is easy to navigate back to if u close the app. I swapped back to the streaming sites and found more NetFlix content immedataly.

Thx for observables though!


Yup like https://www.wepickflix.com :p - I tried to implemented a "group recommendation" - since watching "tv/shows/movies" is many times a "group activity".


I would be a lot more interested in a technical article than hearing about the history of Netflix.


We've recently done a large blog series on experimentation. https://netflixtechblog.com/netflix-a-culture-of-learning-39...

Kinda covers everything and links out to posts that cover the software engineering in more detail and papers that cover the math in more detail.


the blog posts series on rec system is interesting. But when I watch Netflix I still run into simple issues. can you please take care of some basics first, such as if I press 'thumbs down' then Netflix would never recommend that title again or similar to such titles.


They can't do that. Netflix now has very little content since all the big players have pulled out and started their own streaming services. Netflix has to recommend what it has: old movies, subpar TV shows, things you don't like, Netflix-produced content.


You might be interested in Trustworthy Online Experiments by Kohavi et al then. I enjoyed the historical progression!


The Netflix recommendations for me are quite bad. I literally aost always end up watching YouTube.

It makes it is super difficult to discover movies, even if you have pretty good idea about what kind of movie you want.


The only reason Netflix is in FAANG is because without them, people would be saying a very naught word.


The reason they're there is that they are known primarily for the high salaries they pay.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: