
Netflix Data Reveals When TV Shows Hook Viewers - waterlesscloud
http://variety.com/2015/digital/news/netflix-tv-show-data-viewer-episode-study-1201600746/
======
gleb
This study should be an embarrassment to a company that cultivates a
reputation of knowing something about statistics. It's like the last 500 years
of statisticians working on survival analysis didn't happen.

Their conclusions are not supported by the data, the study methodology is
flawed, and their "average of averages" metric would have gotten me an F from
my 8th grade physics teacher.

The key methodology problem is this: _" A hooked episode was defined when 70%
of viewers who watched that episode went on to complete season one."_ What you
are actually want to know is after which episode the retention curve flattens
out. With their methodology they'd claim a "hooked episode" even in case
show's retention followed exponential decay (constant % drop after each
episode) - meaning there is no "hooked" episode at all, and retention curve
never flattens out.

 _" Hooked episodes were first identified by country, then averaged to create
the global hooked episode."_ This is just awful. Again, high-school physics -
don't average averages. I understand what they are trying to do but this is
not how you do it.

And to add insult to the injury they stuck a House of Cards spoiler in the
first sentence.

------
abhv
This is a press release from Netflix (no link to the underlying study).

[https://pr.netflix.com/WebClient/getNewsSummary.do?newsId=26...](https://pr.netflix.com/WebClient/getNewsSummary.do?newsId=2603)

Except they use the data to make stupid conclusions:

"However, in our research of more than 20 shows across 16 markets, we found
that no one was ever hooked on the pilot. This gives us confidence that giving
our members all episodes at once is more aligned with how fans are made."

I don't see how the study sheds light into "causality," i.e. "how fans are
made."

Since netflix does not need to sell commercials, there is no incentive problem
with binging. A network, however, needs to keep your attention every Wednesday
evening to sell their commercial slots, so they have to pace their material
episode by episode.

~~~
spacecowboy_lon
Not sure about that I think many people where hooked on Sherlock (the BBC one)
by the first episode.

~~~
JustSomeNobody
How does one example ever prove or disprove anything?

~~~
bladecatcher
[https://en.wikipedia.org/wiki/Counterexample](https://en.wikipedia.org/wiki/Counterexample)

~~~
ryangittins
A counterexample is only definitive proof when opposing an absolute.

~~~
bladecatcher
I was only responding to the parent question.

------
mmahemoff
"Netflix has no plans to use Big Data to rejigger the way TV shows get made"

It's hard to believe data like this won't end up being used to reverse
engineer some aspects of their content, especially when they are pumping out
dozens of shows. Easy to experiment with some of them.

Amazon already has users voting on solo pilots and it would be very surprising
if they don't end up leveraging their analytics for creative.

~~~
JustSomeNobody
Yeah, I don't really believe it either. There's already formulaic writing
(ideal number of plot twists, etc.), so there is precedent.

~~~
SeanBoocock
Indeed. Netflix has previously cited their use of data analysis in investing
in new shows like House of Cards:
[http://www.nytimes.com/2013/02/25/business/media/for-
house-o...](http://www.nytimes.com/2013/02/25/business/media/for-house-of-
cards-using-big-data-to-guarantee-its-popularity.html)

------
zaphar
Here's what I'd like to know. Do they have any data on the optimum length of a
plot arc for a shows? There are a number of shows I've gotten hooked and then
around about season 5-7 I've dropped because it became obvious they were just
milking it and inventing gratuitous plot extensions instead of ending it
cleanly.

Some shows can go for really long runs because the plot arcs tend to be 1-3
episodes in length. The Law and Order style of show. Anything that goes
multiple seasons in length just seems to lose me after at most 2 plot arcs.
Maybe I just get burnt out?

~~~
ttkeil
> There are a number of shows I've gotten hooked and then around about season
> 5-7 I've dropped because it became obvious they were just milking it and
> inventing gratuitous plot extensions instead of ending it cleanly.

This is what I'm worried about with 'House of Cards.' It feels like they might
have 1-2 good seasons left, but unless Frank suddenly decides to deceive his
way into becoming world dictator, I'm convinced that they need to end the show
in fewer seasons than it already has.

------
fitzwatermellow
I wonder if there had ever been a show in the history of television in which
70% of viewers who watched the pilot made it to the finale. Fortunately, the
article states Netflix has no plans to "rejigger" (i like that word, plan on
using it more) its method of greenlighting shows. Which probably, despite the
enormous quantity of data they collect, still resembles something akin to that
scene in "Network," where they consider doing a docu-drama about the
Symbionese Liberation Army.

You can count me as one of the 30% that never makes it past the first ten
minutes of even the most acclaimed serial content. I just like movies. But out
of curiosity I did watch the extended trailer of the 500 Startups reality show
"Bazillion Dollar Club." There is a scene where a team is finalizing a
prototype with its sourced design consultants. Its all dudes and they select a
shape that is, well, let's just say suggestively masculine in its proportion.
They decide they need to counterbalance their opinion with the female
perspective and invite the ladies on staff to give their opinion. The women
choose a completely different design. Soft, with rounded curves. But most
importantly they have a visceral negative reaction to the men's choice. But
the startup leadership completely ignores what I would consider a very
important data point in favour of an arbitrary metric. That "70% of men make
electronic purchasing decisions for their households." Completely ignoring the
fact that generally, men don't really care about the appearance of their
gadgets, just that they function properly. And that women generally tend to
make the design choices at home. Obviously a very broad generalization but
still.

The lesson here is about letting stats influence faulty reasoning while
ignoring intuition. It doesn't appear that Netflix is allowing data to dictate
programming. Yet.

~~~
pinaceae
any stats to backup your claim that men generally don't care about the
appearance of a gadget?

Seems there are whole industries begging to differ, starting with cars.

------
sveme
I suppose they measure it by looking at the episode at which users continue to
watch the whole series, in contrast to those who stopped viewing at an earlier
episode? There is so much more you could get out of this:

(i) What's the drop off rate for viewers over episode number? For example, I
stopped viewing quite a lot of these shows right before other viewers got
hooked (Marco Polo: episode 2, sense 8: episode 2, that superannoying lesbian
scene, Arrow: episode 3 or 4)

(ii) What's the interval distribution between successive episode viewings,
does it differ between shows, is it predictive of a tendency to stay with the
show (i.e., how bingy is the show, if that's a word)?

(iii) Age and gender distributions for drop off rates

And so much more. Any way of getting my hands on the data?

~~~
sveme
Here's at least the press release with a little more details:

[https://pr.netflix.com/WebClient/getNewsSummary.do?newsId=26...](https://pr.netflix.com/WebClient/getNewsSummary.do?newsId=2603)

\------------------------------------------------

DO YOU KNOW WHEN YOU WERE HOOKED? NETFLIX DOES Netflix Unveils When Fandom
Begins For Some of Todays Most Popular Series Sep 23, 2015 Los Gatos, Calif.,
September 23, 2015 -- It may have taken Walter White nearly an entire season
to become Heisenberg and Frank Underwood 13 episodes to become VP (spoiler
alert!), but it turns out fans committed to these series long before those
plot twists unfolded. Hint: it wasnt in the pilot episode.

Netflix analyzed its global streaming data* across the inaugural seasons of
some of todays most popular shows - both Netflix original series and shows
that premiered on other networks - looking for signals that pointed to when
viewers became hooked. It turns out that when commercial breaks and
appointment viewing are stripped away and consumers can watch an entire season
as they choose, you can see fandom emerge. That is, 70% of viewers who watched
the hooked episode went on to complete season one or more poetically, when
members were hooked and there was no turning back.

“Given the precious nature of primetime slots on traditional TV, a series
pilot is arguably the most important point in the life of the show,” said Ted
Sarandos, Chief Content Officer for Netflix. “However, in our research of more
than 20 shows across 16 markets, we found that no one was ever hooked on the
pilot. This gives us confidence that giving our members all episodes at once
is more aligned with how fans are made.”

While the data identified the hooked episode, it was shy on pinpointing exact
moments __, but we have a few ideas of our own to help jog your memory... For
starters, in Breaking Bad it may have taken the flip of a coin to decide
whether Jesse or Walt would put the finishing blow on Krazy 8, but when the
decrepit heap of a former drug dealer rains down from Jesse’s ceiling, theres
no denying viewers would stay to see how the season cleaned up (episode 2).
Speaking of messes, Crazy Eyes drops both poems and fluids in her roller
coaster romance with Piper in Orange is the New Black, but it was likely the
throw of a pie to defend her (then) bae’s honor that had members asking for
seconds (episode 3). For Dexter another episode equals another body, this time
courtesy of the “Ice Truck Killer,” but our money’s on Dexter’s trip down
memory lane reliving his inaugural kill that was the real tipping point -
after all, fans never forget the first time (episode 3).

“There’s a unique sense of intimacy with creating a show for Netflix. Knowing
you have an audience's undivided attention and that in essence, they are
letting these characters in their home, we unfolded storylines at a more
natural pace,” said Marta Kauffman. “In episode four, we see Grace and Frankie
having no choice but to confront their fear, anger and uncertainty head on,
which to me as a creator was a nice turning point to shift the narrative to
focus on the future instead of the past; it is nice to know viewers were there
right along with us.”

While around the world the hooked episode was relatively consistent, slight
geographic differences did present themselves. The Dutch, for instance, tend
to fall in love with series the fastest, getting hooked one episode ahead of
most countries irrespective of the show. Germans showed early fandom for Arrow
whereas France fell first for How I Met Your Mother. In Better Call Saul,
Jimmy McGill won Brazilians over one episode quicker than Mexicans. And Down
Under, viewers prove to hold out longer across the board, with members in
Australia and New Zealand getting hooked one to two episodes later than the
rest of the world on almost every show. Despite these differences, the hooked
moment had no correlation to audience size or attrition, regardless of show,
episode number or country.

Methodology: The data in this research was pulled from accounts who started
watching season one of the selected series between January 2015 - July 2015 in
Brazil, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, Mexico,
Netherlands, Norway, Sweden, UK and US and between April 2015 - July 2015 for
Australia and New Zealand. A hooked episode was defined when 70% of viewers
who watched that episode went on to complete season one. Hooked episodes were
first identified by country, then averaged to create the global hooked
episode. The hooked episode had no correlation to total viewership numbers or
attrition.

 _Denotes shows where for one or more countries, the show was unavailable to
watch on Netflix and therefore the average is comprised of data from less than
16 countries

_ *The Netflix research didnt indicate exact plot points, but it did confirm
episodes.

~~~
IanCal
> Frank Underwood 13 episodes to become VP (spoiler alert!)

Spoiler alerts _after_ the spoiler is like putting a warning sign at the
bottom of the cliff.

------
netcan
There's some side of me that's unhappy to see stats (even as benign as this)
being considered. It's not really the idea that statistics will influence
content as much as it is about who and how it will influence it.

I've been reading "A Song of Ice and Fire" (the source books for "Game of
Thrones"), like a lot of people, for years. That is, I've been reading a book
enjoyably for a couple of months and then waiting 5 years for the next one.
It's an incredibly intricate and big story spanning a lot of character and
geography. It's very big. The author, is obviously a special talent with
incredible story telling ability and character perspectives. The talent for
getting into characters' heads makes me think he must be a very emotionally
invested artist.

Anyway, among all the characters, story, best seller pressure, TV show, and
such writing the books takes a long time. He's got a Ford Prefect like
relationship with deadlines.

Now.. in my (naive) mind, that's as it should be. A chaotic, burdensome
process that (at the end, sometimes) produces art.

TV shows and films are the art of our times. This is not a statement about
quality, just about the reach, influence and such. The people who make films
are famous and admired. Everyone gets the references and allusions. Almost
everyone indulges in it, grows up with it. A lot of people here probably got
my Douglas Adams reference, but probably fewer than would have gotten a
reference to "Morpheus," even in this biased crowd.

But, movies and television shows are made by big groups of people, money,
risk-capital, adult supervision. I'm OK with statistical information about
pace, plot or whatnot eventually permeating through the arts. That probably
makes the arts better. I dislike the idea of it being used as a mallet in the
proverbial committee designing the thing.

I like sausage. I prefer sausage to be made in a nice way. If you insist on
both making it in a way I don't want to see and constantly reminding me that I
should not look into how it is made… I'll have some beef stew instead.

------
sweezyjeezy
Seems a bit weak - I don't think you can really drill down onto one episode as
one that 'hooks' a viewer, rather than all the subsequent episodes being
excellent. At least, this article didn't convince me.

~~~
zeckalpha
Think of the season like a funnel. What's the drop off for each episode?
What's the likelihood if they've made it this far they'll watch the rest of
the season?

~~~
sweezyjeezy
...right, but then I'd rather see the funnel than which episode gets you to
70%, because I don't think without that context you can really say anything
about the episode.

------
wcummings
My rule of thumb is to watch at least 3 episodes, because anecdotally the 3rd
episode is when a lot shows hook you.

------
justinlardinois
> “This won’t have any direct effect on the creative process for our
> showrunners/creators,” a Netflix rep explained.

Even if Netflix doesn't force it on production teams, you know someone
somewhere down the chain is going to have this in the back of their head.

------
fahimulhaq
Highly unlikely but would be interesting if Amazon releases the same data for
ebooks. They might already be doing stuff like showing those pages in the
preview that have the highest chance of getting the reader hooked.

~~~
mattmalin
Kobo evidently are doing some analysis on completion rates, and published a
report last year aimed at publishers using analysis based on similar metrics:

[http://cafe.kobo.com/_ir/159/20149/Publishing%20in%20the%20E...](http://cafe.kobo.com/_ir/159/20149/Publishing%20in%20the%20Era%20of%20Big%20Data%20-%20Kobo%20Whitepaper%20Fall%202014.pdf)

Further numbers into completion rates split by region and genre also from Kobo
made their way into this nytimes article earlier in the year:
[http://www.nytimes.com/2015/02/05/arts/international/keeping...](http://www.nytimes.com/2015/02/05/arts/international/keeping-
tabs-on-best-seller-books-and-reading-habits.html)

I'd be interested in the raw data, particularly also from Amazon based on
Kindle ebook read completion though I imagine that won't get shared.

~~~
fahimulhaq
Nice find. Yes raw data will be more useful. Especially, with Amazon's Pay-
Per-Page for Kindle Unlimited, if such data is available, it will be
interesting to see how authors reverse engineer this process to achieve
maximum page views (and completion rates)

------
wtbob
I wonder if knowledge of this might have saved Firefly…

------
clentaminator
Did Netflix only analyse shows with more than one season? I'm surprised Fargo
wasn't included as it had me hooked from the first episode.

------
yashafromrussia
Pretty good list of tv shows to watch :P definitely watching the shows that
you get hooked onto from the 2nd episode

------
greatthanks
I am _absolutely_ not at all whatsoever impressed with Netflix' data science
capabilities.

They waste one row for:

\- watch it again ...

\- they suggest stuff based on what I added to my list, but didn't watch yet

\- they suggest movies based on what I started to watch but didn't finish - as
if that wasn't an indicator for that I didn't like it

\- ...

All those companies suggesting on varies channels how nifty and smart they
work with their _big data_ \- and at the bottom line they just fail on
actually improving something.

~~~
tfgg
Do you think that those design decisions are not backed by data demonstrating
better engagement levels versus other interface layouts? I'm not saying that
their interface is perfect, but I don't see how you can make sweeping claims
about their data science capabilities based on your one data point of
disliking their interface.

~~~
fluidcruft
I think at some point people assume others are more competent than they
actually are. I mean, honestly, here you are defending Netflix based on
nothing more than stereotypes and projecting competence into them--when the
linked PR story is so ridiculously methodologically flawed that a rational
observer would seriously question their basic mathematical and psychological
literacy.

