
Netflix’s Secret Special Algorithm Is a Human - donohoe
http://www.newyorker.com/business/currency/hollywoods-big-data-big-deal
======
JeremyMorgan
Hollywood Video's corporate office had a guy, one guy whose job it was to just
watch movies. He'd watch them, make sure it was categorized properly, then
create associations with other movies that "you might like" if you like this
movie, and vice versa. All kept in a spreadsheet that we later plugged in.

We also thought there should be an algorithm, but he was pretty dang good at
what he did.

~~~
boomzilla
It's simple ROI. There is not that many movies this is a task that one doesn't
need to scale with computers. Hiring a guy is likely much cheaper than
investing on the software. Plus I am sure there are plenty of people who would
like to watch movies for their day jobs.

On the other hand, I am pretty sure there is at least some degree of automated
video classification at Youtube.

~~~
RogerL
I think it does need to scale.

What happens when after a bus event? Someone else has to watch every film in
existence? It is not enough to just watch the new films - you have to have a
memory of all other films in order to make that association.

~~~
gwern
> Someone else has to watch every film in existence?

To some extent, all the film program students and film critics provide your
backup reserves: they're watching tons of films on their own, and you don't
even have to pay them until you hire them.

~~~
wallflower
> VidArc was a tiny store in a mini-mall, with barely room to squeeze past
> other customers, but it made up for its size with the percentage of rare and
> obscure titles that were available, and with the knowledge of the film-nut
> staff, notably the cinema-obsessed and mile-a-minute talker Quentin, whose
> low-budget life at the time has been explored in several books. Denise's
> card was number 1410, and when I made my trek from Oregon in the Orange
> Monster (my '72 Olds Cutlass), I became a customer as well, discovering the
> world of strange and disturbing cinema under the tutelage of Quentin,
> Rowland Wafford, Gerald "Big Jerry" Martinez, Stevo Polyi, Roger Avary, and
> the owner of VidArc, Lance Lawson.

[http://toddmecklem.com/quentin.html](http://toddmecklem.com/quentin.html)

------
scott_s
In other realms, we don't call this _data_ but _intelligence_. Humans will
still make the decisions, but in more and more places, the decisions are more
likely to be informed by data (intelligence).

Consider that Netflix's data is tiny compared to the amount of data that any
government must sift through. Algorithms don't make government decisions,
people do. But they (hopefully) base those decisions based on intelligence
(some of) which was gathered and filtered by algorithms, then synthesized by
humans.

~~~
SquidMagnet
I am not entirely sure what you are saying here, but the way I have always
understood it is: data is knowledge; the ability to apply knowledge is
intelligence.

Netflix has a considerable amount of data (knowledge) and its algorithms
exemplify some efforts to apply that knowledge (intelligence). As it stands
presently, though, humans still tend to be more intelligent than any
algorithms we have created. (Generally speaking, of course.)

I think we are trying to say the same things here, right?

~~~
scott_s
I'm trying to reframe what Netflix does as something that is already done - we
just tend to use different names for it in those other domains. What's new is
that we're doing this old thing - processing massive amounts of data,
extracting the relevant facts, synthesizing those facts into a coherent story,
and presenting that story to decision makers - in new contexts.

In the Netflix context, what is mostly called "data" is called "intelligence"
in, say, government decision making.

------
raverbashing
As much as people (especially here) think that everything can be solved by Big
Data/Machine Learning, human experience and intuition are still very
important.

No, you can't A/B test your new logo, or the design of your page from scratch.

Big Data can't tell you your product sucks. Option A may be better than Option
B but this is in the context of both options (and not considering all other
possibilities)

I see companies firehosing every tiny bit of consumer data hoping to be able
to make sense of it all and find something there. Meanwhile they're missing
whatever their competitor is doing and what their consumers are liking about
it.

~~~
jobposter1234
Those companies are just cargo-culting into the Big Data cult, like everyone
else. It doesn't mean they aren't getting useful answers from their black box.

Heck, sometimes it's even useful to have a Magic 8 ball make a decision for
me.

------
wallflower
See also Shazam's Secret

> The hunt keeps Mr. Slomovitz on his toes. Every morning, he skims dozens of
> music blogs, checking for new releases he might have missed, as well as the
> iTunes, Amazon.com and Billboard charts, and blog aggregators like the Hype
> Machine.

Most weeks he also goes to local record stores to see if there is something in
stock he has not heard of, or if older albums are being remastered or
reissued. And he listens to local radio stations, especially near
universities.

"Shazam’s Search for Songs Creates New Music Jobs"
[http://www.nytimes.com/2011/02/14/technology/14shazam.html](http://www.nytimes.com/2011/02/14/technology/14shazam.html)

~~~
skrebbel
Shazam actually has 2000 music geeks in a call center who listen to tiny audio
fragments and guess which song it is. The algorithm just makes sure the rock
geek gets the rock songs, and so on.

~~~
afro88
Really? I'm genuinely interested in this. Do you have an article or interview
or something with more info?

------
ljd
I see this argument akin to something along the lines of, "A website is really
a human because someone needs to maintain it and make decisions on it's
appearance."

You are describing a tool. People don't say hammers are humans, either.

Netflix's special algorithm is indeed an algorithm.

~~~
abandonliberty
I was really hoping Netflix had a secret warehouse filled with hundreds of
workers whose job was to provide recommendations based on what you like, a la
'invisible boyfriend' [1]

Sadly, they just have people on the quest for the perfect hammer.

[1][http://www.washingtonpost.com/news/the-
intersect/wp/2015/01/...](http://www.washingtonpost.com/news/the-
intersect/wp/2015/01/22/i-paid-25-for-an-invisible-boyfriend-and-i-think-i-
might-be-in-love/)

------
otakucode
It always amazes me how the people in the content creation industries keep
their jobs. Even though comprehensive studies have shown that the public
responds randomly to entertainment products, they still skate by claiming to
'know what the public wants.' Analysis of the decisions of past executives at
movie companies showed that executives who were said to be "on a hot streak"
were mostly benefitting from the results of projects started by their
predecessors, and then when their "streak" ended and they left, their
replacement got the same benefit from the projects they left in the pipeline.

The book 'A Drunkard's Walk' analyses various different industries and
situations and shows where randomness shows up. The success of media is one of
the strongest ones. No factor correllates with success. Not budget, star-
power, genre, directors, nothing. For every runaway hit there are exactly as
many abyssmal failures. Before Titanic hit theaters, film critics and industry
insiders were dead certain that it would prove to be the most monumental
theatrical failure in history, making Kevin Costner's Waterworld look like a
walk in the park. Of course, they were wrong. They will always be wrong as
often as they are correct. Their entire careers are built on, quite literally,
absolutely nothing. These executives choose what gets produced, and they are
incompetent at it. Yet they are paid millions of dollars. It's astonishing
that they get so far with unmitigated bullshit.

This is why large movie production houses will die. Their performance has
always been random in terms of producing successful content. But they always
made up for it by having total control of the distribution. Now, distribution
is worthless. Any 12 year old with an Internet connection can distribute media
better than large corporations can. Take away that control and profit from
distribution, and those companies will end up simply fading into bankruptcy
after enough failed projects pile up.

~~~
roasm
Large movie production houses won't die in the near future, because predicting
a financial winner isn't as hard as you make it sound. We currently have a
movie economy that is driving the big producers to make big budget comic book
sequels that do well internationally (generally based on the universal appeal
of liking suspense and action). They've boxed themselves into a very
profitable corner of funding high budget movies that independent producers
can't fund and then profiting internationally. They make a movie for $200M
then make $300M+ in sales.

I do agree that more independent (but still highly funded) producers are
rolling dice, then manufacturing winners with their marketing power.

------
stillsut
I see how netflix's dataset helps them produce an addictive serialized drama
like House of Cards. There are many examples to learn from, and the shows are
basically built out of a formula.

What I don't see is how this would help you make a critically well received
documentary about Nina Simone. Sure, you might be able to predict a lot of
people will stream it, but how does that make it well received at Cannes?
Nothing in their dataset is telling them about the art of film-making.

~~~
smacktoward
The argument of the article is that Netflix's dataset _doesn 't_ really help
them make decisions like what the content of their next documentary should be.
It argues that what happens instead is that Netflix's programming honcho, Ted
Sarandos, picks projects based on the person who's putting them together,
looking for creators with a solid creative track record and an existing cult
following:

 _> I began to sense that their biggest bets always seemed ultimately driven
by faith in a particular cult creator, like David Fincher (“House of Cards”),
Jenji Leslie Kohan (“Orange is the New Black”), Ricky Gervais (“Derek”), John
Fusco (“Marco Polo”), or Mitchell Hurwitz (“Arrested Development”)... I do
think that there is a sophisticated algorithm at work here—but I think his
name is Ted Sarandos._

So take the case of the Nina Simone doc. It was directed by Liz Garbus
([http://en.wikipedia.org/wiki/Liz_Garbus](http://en.wikipedia.org/wiki/Liz_Garbus)),
whose previous films have been Sundance- and Oscar- fare for a few years now.
So it could just be that Sarandos looked at her track record and decided that
Garbus was an up-and-coming talent, and that mattered more than the specific
content of whatever her next project was going to be. He was just betting that
a Liz Garbus documentary was going to be great, no matter what story it told.

------
karmacondon
The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't
"Humans are good at some things and computers are good at others". It's that
there isn't a good reason to make the investment in solving this particular
problem algorithmically (yet).

Algorithms are great when you need scale, especially in situations where a 10%
improvement in prediction accuracy can make a big improvement in the bottom
line. Netflix and other studios might greenlight several new shows in a year,
out of dozens that receive consideration. And the Pareto Distribution is in
full effect. Most of the profits and awards come from one or two big hits.
Algorithmic decision making just doesn't make a lot of sense in situations
with a small sample size and uneven reward structure.

It doesn't mean that it isn't possible, though. If someone were to make the
massive investment necessary to do a more thorough analysis of the content
creators, the actors, the scripts and potential audiences and all of the other
possible inputs then algorithms could probably do as good a job as humans, if
not better. Netflix and others have only taken baby steps in this direction,
working with data that is readily available and using predictive techniques
that are well tested and understood. Given the nature of the problem, it
doesn't make sense for them to approach it any other way at this time. But
when it comes to making billions of recommendations to millions of people per
day, they still rely heavily on data and algorithmic prediction. There's a
time and place for everything. The time and place for algorithms in our daily
lives is changing and expanding, but very slowly.

------
rurban
I also rather trust the algorithm to see what is a good movie and what is not.
In my case the collected ratings of the experts, not the users. Current
example: [http://cannes-rurban.rhcloud.com/Sundance2015](http://cannes-
rurban.rhcloud.com/Sundance2015)

There are some outliers, typical "festival hits", which only relate to
festival specialists, but they are easily detectable, with a "10%" human
bullshit detector. Like last years Godard at Cannes, 2012 Leos Carax, 2011 the
Kaurismaki and 2010 both the Godard and the Jury winner Weerasethakul. Those
outliers are even statistically detectable.

One thing is clear, you can trust the collected experts more than the juries.
So I can fully confirm the story.

~~~
abruzzi
The problem I've found is the algorithms for detecting taste seem to struggle
greatly the further ones taste deviates from the norm. I find Netflix's
recommendations to be fundamentally useless and is always reccomending things
in which I have zero interest. But I've also found Rotten Tomatoes to be
borderline useless as well (the second worst film I've seen in the last decade
--Snowpiercer--has a very high RT score...go figure.)

~~~
keypusher
Do you actually rate the things you watch on Netflix? Or go back and rate a
bunch of popular movies you loved and hated? Because if you don't, you will
just keep getting the same mainstream recommendations.

------
redRobo
The title is completely misleading, seems like it's talking to the
recommendation feature on Netflix, but it isn't. Everyone with common sense
knows that big executive decisions aren't made solely by computers, yet the
author seems amazed that big executive decisions are ultimately made by big
executives. Talk about clickbait, huh?

------
dtjones
>Even Google, the champion of algorithms, employs substantial human
adjustments to make its search engines perform just right.

The author doesn't have a solid grasp on machine learning.

The 'human adjustments' provide feedback to the algorithm, which the algorithm
then uses to update and improve performance. His tone implies its a bad thing
to use human feedback.

~~~
mostly_harmless
Ideally, it would be a completely automated system. You don't need to
understand machine learning to know that it would be better for the computer
to do everything itself without human intervention

~~~
CHY872
How so? If the cost of making the computer able to do everything itself is
greater than the cost of an equivalent system that takes human input, then you
should use the human input version every time.

~~~
diminoten
Monetary cost isn't the only cost.

Also, humans are prone to error and many other inefficiencies. They get sick,
quit, fluctuate in performance, require attention and care... you say "cost"
like it's some easily calculable number, but it's a whole boatload of
intangibles that just... disappear if the computer does it.

------
j2kun
> featuring the television star Melissa _Raunch_

Whoa there! It's _Rauch._ Kind of a Freudian slip there, considering the
actress is being named in relation to a sex scene.

~~~
ScottBurson
Here's another:

> Television studios have Nielson ratings

It's "Nielsen".

~~~
j2kun
Looks like they've fixed it. But seriously don't they have editors reading
this stuff before it goes live?

------
hotgoldminer
Dao Nguyen @Buzzfeed, "Data should not dictate your strategy," Nguyen says,
"But you should understand what data tells you and also what its limits
are."[0]. Heard this a few weeks back and still resonating with me.

0 -[http://www.marketplace.org/topics/tech/buzzfeed-wizard-
who-c...](http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-changed-
media-we-know-it)

------
domdip
What a weird strawman article. Did the author really think Netflix produced
television shows in the same way that Siri suggests gas stations? Any
blockbuster-finding algorithm (or any other component of this decision) is
going to have false positives and require human intervention.

------
ctdonath
Drive such a thing thru Amazon Turk perhaps?

Might just be the "killer app" for Turks: get paid to watch movies!

~~~
sireat
MTurk has already been used to classify and tag videos.

I spent a month as Turker (as an experiment) a few years ago.

The best paying HITs were provided by a porn company to classify and tag their
videos. You were given a sequence of images from the video presumably to save
bandwith.

I even debated scripting a helper tool which would take a few keystrokes (for
example dvda) and generate a suitable description.

------
minthd
This article looks at single point in the industry with regards to automation.
But i believe the important point is whether the addition of data provides
more efficient movie production, with less failure , and by how much.No need
to automate a movie you don't produce.

------
zkhalique
In other news, Pandora is built on human input, among other things. The Music
Genome Project.

