
Netflix: What Happens When You Press Play? - yarapavan
http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html
======
jedberg
> I pulled this chapter together from dozens of sources that were at times
> somewhat contradictory. Facts on the ground change over time and depend who
> is telling the story and what audience they're addressing.

You know old time journalists had a solution to this. Interviewing subjects
with first hand knowledge.

I know I rag on highscalability a lot, but I feel like they deserve it.
They've written multiple articles about topics in which I would be a first
person resource (Reddit and Netflix), and not once have they ever contacted me
or anyone I worked with for an interview. They've _quoted_ me from
presentations, but the quotes were out of context and wrong they way they
quoted it.

All of this could have been cleared up with a single interview of anyone who
is a primary source expert.

~~~
toddh
Hi Jeremy,

I'm very sorry if I've misquoted you or quoted you out of context. I'd
certainly correct them instantly if contacted. I've searched my email and I
don't see any corrections, so I'm not sure how I can make amends.

I pull a lot of quotes from a lot of sources. Many thousands a year. My goal
is always to highlight ideas that I think might help readers do their job
better. Highlight is they key word here. That you say those sort of things
often is not surprising. My hope is always readers will be made curious enough
to follow the link and learn more on their own.

And unfortunately there is no "they" at HighScalability.com. There's only me
and there's only ever been me. So all mistakes are mine and only mine. There's
never been much money or time to be as comprehensive as I'd like.

I do the best I can, but I often fall short, no doubt. I'm not a journalist. I
never said I was. I'm a programmer. Have been all my life. I started this blog
a long time ago. It was just stuff I found interesting. For whatever reason it
got popular for awhile so I tried to make something out of it. That, like most
of the industry is petering out now, so we'll see what the future holds.

The point about interviews is a good one and one I've wrestled with. Over the
years I've conducted _a lot_ of interviews. Much less so lately, and that's
for a reason. I've learned that presentations at major conferences are usually
the best resources. Presenters put a huge amount of effort into making those
talks. And everything they say is open and approved for public consumption by
definition.

Interviews with someone like me are often a bother, a time sync, so I don't
usually get as much out of them. Interviews have a high overhead, taking a lot
of time to setup and there's often a lot of red tape and legal wrangling that
must go on to get access and content approval. Which is again, why,
presentations are such awesome resources.

This article was derived mostly from people working on a project making a
major presentation or from primary sources writing an article. I couldn't do
better than that.

And regardless, I would still make mistakes. Believe me. It's surprisingly
hard to make sense of an interview and turn something from thread mode into
document mode. That process is an error prone one because I have to fill in
the gaps and there are always gaps.

It's rare that I've ever just published an interview. I don't think they are
that useful in general. I want people to read a highly condensed document they
can get something concrete out of with minimal effort or just ignore easily if
it's not relevant to them. The strange style I've developed reflects that
goal. This article was of course different because it has a different
audience.

So again, I apologize. But I guarantee my mistakes are never intentional,
never for money, and never for lack of caring or effort.

~~~
stdclass
Just wanted to chime in and say thank you! I've enjoyed reading your blog a
lot!

~~~
toddh
Thanks, much appreciated!

------
curun1r
> The advantage of having three regions is that any one region can fail, and
> the other regions will step in handle all the members in the failed region.

At a previous employer, we met with AWS engineers who told us that this sort
of thinking wasn't correct. Availability zones are for failure tolerance.
Regions are conceptually different and for things like moving compute closer
to clients, mitigating political instability or ensuring that data is held in
a certain jurisdiction. To back up that claim, they told us that amazon.com
ran out of a single region, us-west-2.

As an aside, another interesting point from that meeting was that autoscaling
wasn't a good strategy for scaling up in the event of the loss of an
availability zone. The reason why is that in those scenarios, a lot of other
customers will be trying to fail over as well and autoscaling can get very
slow. Instead of autoscaling, they recommended that we run 150% of our per-AZ
capacity in 3 AZs so that if an AZ went offline, the remaining two would
already have the necessary capacity. We still used autoscaling, but it was for
responding to spikes and troughs in our own load, not failure scenarios.

I'm curious whether these two observations are correct since it seems like a
lot of people view regions and autoscaling through a failure-recovery lens.

~~~
mimerme
I think this is another case of "your not really supposed to...but it works
¯\\_(ツ)_/¯"

For your first point, I think it depends on your usage scenario. With a big
company that requires good performance to satisfy their users, like Netflix,
having multiple regions is a given. But what the article probably missed is
that they also most likely use AZs within the same region to mitigate
failures, but since they have multiple regions at hand why not use them to
mitigate failures as well? And their monthly tests are a simulation of a worst
case scenario if an entire region goes down.

As for autoscaling I also think that people think of it as a magic solution.
More users? Autoscale. Zone down? Autoscale. But the problem arises when they
think that it's instant because it isn't. Things have to happen behind the
scenes, but for many people, they only care about what happens on stage, so
for them configuring autoscale is just better because it's easier than load
balancing on 3 different AZs.

------
temp
I'd be interested in finding out what happens when I press the thumbs down/up
icon on Netflix, because the match ratings that are displayed to me seem to
have nothing to do with either my ratings or with my watching history.

~~~
juanmirocks
It records your like/dislike, but it doesn’t distribute the change to its
database immediately. Same for YouTube or so many other services.

~~~
temp
The thing is I don't feel like any of the match ratings in the last year and a
half since I've been using Netflix have had anything to do with my interests,
the ratings I've given out or with my viewing history.

Unless propagation to their database takes two years, I'm really interested in
how they actually arrive at match ratings.

~~~
syshum
I believe largely because a single "like / dislike" is massively inferior to
the old traditional Rating system where I am able to Rank content I like...

------
forgotpassagan
I love how Netflix is so open about their architecture, but I have one strong
criticism.

After trying to get Spinnaker running, which is like 10 services for a fairly
straightforward application, and reading about how Netflix has Cron jobs
running to 'clean up' data inconsistencies... I started to think Google has a
better approach to scaling the code.

Netflix is strong on eventual consistency, and making services as small as
possible. Having tried to build something in that image, holy shit the issue
of distributed transactions becomes a nightmare.

We had to build error handling in everything to handle the failure of
everything else. Maybe half our code was 'just in case' to deal with exploding
failure scenario complexity during service call fanout. If your service calls
out to two or more services that also do data updates, God help you. There's
no call ordering or way to make sure both succeed or fail, so if one does you
could easily end up in an invalid state.

Contrast this to Google's approach. Maps is one service, Docs is one service.
Their service boundaries are much larger so they can shove the complexity of
consistency and rollbacks back into the DB where it belongs. And they avoid
eventual consistency as much as possible.

If Google can make these big 'monoliths' work at huge scale I don't see any
advantage to 'microservices', it's just a bunch of pointless overhead. I think
Netflix has had a little too much industry koolaid ...

~~~
saryant
> Netflix is strong on eventual consistency, and making services as small as
> possible. Having tried to build something in that image, holy shit the issue
> of distributed transactions becomes a nightmare.

Distributed transactions are a sure sign that your service boundaries have
been incorrectly drawn.

~~~
forgotpassagan
Not according to Google :). They actually support and encourage distributed
transactions across boundaries.

Imagine all the data Google stores with your user profile across their
hundreds of application boundaries. To keep things consistent you would either
need to store all profile data in one massive service or support distributed
transactions.

The Netflix model is fault tolerance, with every service supporting various
failed profile update scenarios. Google just decided to add distributed
transactions support to spare all that overhead.

------
Abishek_Muthian
I feel bit sorry for Netflix, for the lengths they have gone to get their high
performance/reliable architecture but the clients are still at the mercy of
the OS, browsers, CPU (decoding codecs) of-course assuming high speed
Internet.

From their website -
[https://help.netflix.com/en/node/23742](https://help.netflix.com/en/node/23742)

Resolution:

Google Chrome Up to 720p on Windows, Mac, and Linux. Up to 1080p on Chrome OS.
Internet Explorer up to 1080p. Microsoft Edge up to 4K _. Mozilla Firefox up
to 720p. Opera up to 720p. Safari up to 1080p on Mac OS X 10.10.3 or later._
Streaming in 4K requires an HDCP 2.2 compliant connection to a 4K capable
display, Intel's 7th generation Core CPU, and the latest Windows updates.

P.S Interesting to note that Chrome OS plays 1080p, I assume because it has HW
acceleration enabled by default (in browser alone, not in android apps
though).

~~~
jmiserez
Is it browser limitations, or just DRM limitations? Youtube seems to do 4K
just fine.

~~~
Abishek_Muthian
I think it has to do with HEVC h.265 codec & HW acceleration support available
to the browser in the respective platforms.

[https://caniuse.com/#feat=hevc](https://caniuse.com/#feat=hevc)

Not sure why Chrome doesn't support HW for it though albeit Chrome OS. My
Celeron N3060 Chromebook can decode it.

I read Netflix has started using VP9 for streaming, newer CPU's have VP9
decoders in them.

~~~
izacus
No it's actually a DRM limitation. You're not allowed to watch high definition
content of your platform doesn't support latest Widewine DRM even if you have
the HW or SW decoding capability.

------
Walf
I know what it does when I _don 't_ press play: fills my memory with a
boatload of bloated interface that has the shittiest navigation of any media
player I've ever used. And I still can't just view a list of all series/movies
by date.

~~~
nabeards
Which media player has an interface you like? I'm usually an outcast for it,
but I actually like the iTunes Movies interface myself.

~~~
Nala_Alan
I like mpv the most. It's simple, but still has everything you need (and you
can do rest with the keyboard shortcuts), and it's pretty. In my opinion,
however, the problem with Netflix player, is not its design. It's that it's
pretty laggy, it is not nearly as smooth as for example YouTube player.

------
pavs
Many of the things Netflix does here google does too and has been doing for a
long time like GGC (Google Global Cache) for free at ISP NOC or datacenters -
their GGC nodes are many orders of magnitude time is distribution.

90% of the Google traffic through these devices are youtube videos.

They mention it in the article like Netflix pioneered this idea with open-
connect. If anything it's harder to get OC devices from Netflix then it is
from Google.

Even Akamai does have a similar program where they let you put their CDN on
ISP NOC.

~~~
amelius
That sounds a bit like a violation of net neutrality avant la lettre.

~~~
virtuallynathan
It's good for everyone -- if you are an ISP with 35%+ of your traffic coming
from Netflix, you might be begging for these boxes in your network.

~~~
pavs
Yup, as an ISP owner, I can tell you 70-75% of my traffic come through GGC,
for every 1gbps traffic from cache nodes 100mbps on average is pulled from the
internet. Even with GGC nodes, it's still very expensive, BW-wise.

~~~
PuffinBlue
I saw another comment on here in the distant past that said an ISP owner
decided to pull out the GGC boxes because they would sync vast amounts of data
(largely YouTube IIRC) and it wasn't worth it bandwdith wise for them.

It sounds like 10:1 is a decent saving but still objectively a lot of data.

~~~
pavs
You can't survive as an ISP if you don't have GGC nodes or directly peered
with google DC. If your customer wants to watch youtube videos and you serve
them through internet only, which will either mean you will have to buy more
BW (not financially sustainable) or you will have to cap youtube traffic
resulting in inferior user experience and a customer leaving your service. Or
just ban youtube - might as well stop running an ISP.

The best solution is to peer with their DC - let them do the heavy lifting and
you can pull from their cache, then you can chuck the GGC nodes. Google is
building a small DC close to our NOC - which means lots of saving for us and a
better experience for our customers. We can see us chucking the nodes or in
this case, we most likely have to give it back to them or something - they are
very expensive and large supermicro servers.

10:1 is not a lot if you are serving a lot of traffic. For 50gbps, thats 5gbps
bw - that's a lot of bw for free service - we don't cap youtube traffic (users
likely to watch high res 1080p, 4k videos nowadays), we can't sell youtube
traffic (license agreement from google) - so as far as ISPs are concerned its
pissing away money. To be honest services like youtube, facebook and Netflix,
even with their on-premise cache solution are a huge headache and expense for
ISPs - it's not just the BW, rack-space, distribution switches, power, they
all add up.

There is a reason why Google with all their expertise, knowledge, money and
influence can't even make a dent in their ISP business - because even with all
their money - it insanely expensive when you try to cover most of the country.

~~~
foepys
> You can't survive as an ISP if you don't have GGC nodes or directly peered
> with google DC.

Believe me, you can. Deutsche Telekom, Germany's AT&T and Europe's biggest
ISP, does this and in the evening YouTube videos regularly stutter on FullHD.
But the customers just don't care enough.

~~~
pavs
If all the big players do it, you really don't have a choice.

------
kzzzznot
> The video comes in a high definition format that’s many terabytes in size. A
> terabyte is big. Imagine 60 stacks of paper as tall as the Eiffel Tower.
> That’s a terabyte.

What... No. An HD ProRes will be 150-200GB for a 1.5-2 hour film. That is a
1920x1080 Apple ProRes 422 HQ @ ~190Mbps which is the 'gold standard' accepted
Mezz file for Netflix, Google Play, iTunes, AIV (Microsoft, Sony, TalkTalk
etc). There are different codecs for broadcasters but this is the standard for
VoD. If you deliver uncompressed (uncommon, few relative gains) then it might
run into the hundreds of GB but never a multi-terabyte file for a feature.

Although most of this article has been informative, I am dismayed by the
outright BS. It basically ruins any trust I would have of this otherwise well
written piece. Can your exaggeration be explained or justified please?

------
verletx64
There's something about this article, it kind of feels like a sequence of
staccato thoughts. It feels like it kind of jumps around.

~~~
toddh
I did try to tell a story, but I don't think I succeeded as well as I would
have liked. It definitely does jump around more than it should. It's hard to
cover the important bits and link them seamlessly together. I'm just not that
good a writer.

~~~
verletx64
Practice makes perfect! It was frustrating because I think I saw what you were
going for and it's a good idea.

Don't be afraid to drop things, even if they're important.

You've got a title, right? "What happens when you press play". Answer that
question. There's a whole load there that is interesting but doesn't answer
the question of "Given Netflix have a bunch of video ready to go, how does
that reach me?". Don't get me wrong, the rest is interesting, and important
when thinking about Netflix as a whole, but ultimately I'm unsure they fit in
the article.

You might be worried about missing important bits, I personally worry about
losing readers. It doesn't matter how important something is, if my target
audience quits 15% of the way through the article.

Anyway, no sweat really, blogging is hard enough. It's been a long time since
I've written regularly, and I just started a personal blog, and yeah, a few
weeks later im sitting on a mound of drafts I need to take a 'minimum viable
post' eye to. :)

Merry Christmas!

------
twic
_This article is a chapter from my new book Explain the Cloud Like I 'm 10._

The normal expression is "explain like I'm 5", but in the cloud, you need to
double everything, in case an AZ goes down.

------
erstap
I feel like this article could be cut in half if it had been given a sharper
focus. I don't care why I am seeing a specific image for a film and the
reasoning for image personalization. Save that for a different article. I just
want to know what happens after I click play.

------
bendbro
It ticks to 25% loaded even though your internet is down.

------
tw1010
I guess the discrepancy between what I expected the article to be, based on
the title, and what it actually was is a good reflection on the fact that
development of these types of services is no longer built from first
principles, low level to high level, but is rather constructed like a building
from a set of abstracted lego pieces.

------
nvr219
All this tells me is that Netflix is way too powerful a force in my life.
Yikes.

------
dominotw
> Netflix Saves Money In AWS

How much of it due to special discount that netflix gets from AWS. Would a
company thats not highly visible as netflix save the same amount of money ?

> Recommendations.

I find netflix recommendations pretty bad. Does it really work for other
people?

~~~
donarb
It's not visibility, it's how much you spend/use. I'll bet there are large
companies out there you've never heard of that get just as much of a discount
from Amazon as Netflix does.

~~~
qaq
I bet very few do I remember watching someone high up from Netflix presenting
at a conference he quoted that they get 70% discount.

------
thinkMOAR
If they could please make it so that you can search on <any> subtitle
languages, it would really help when the in-laws visit.

------
tlholaday
Author uses acronym CDN without first defining it. Explain Like I Do Not
Already Know What You Are Explaining.

~~~
jordache
that article is pretty technical, beyond the level of technicality of simply
knowing what a CDN is.

~~~
JustSomeNobody
But even technical people sometimes don't know simple acronyms. It doesn't
hurt to define them anyway.

~~~
jordache
mentioning cdn would be akin to a car review magazine mentioning ABS in an
article and then dedicated a couple sentences explaining what ABS is

~~~
yesenadam
Hmm I dont know what CDN or ABS are.

~~~
BoorishBears
Content Delivery Network

Anti-lock Braking System

Both are exactly what they sound like

~~~
yesenadam
OK thanks :-)

------
inciampati
Nothing. So you go to your nearest torrent aggregator and are streaming it
down in 3 minutes.

~~~
HenryBemis
I read the word "nothing" and I thought to mysel: now THIS will turn
interesting!!! (thinking that this will point to removal of Net Neutrality and
the risks and conflicts it will bring to online services once ISPs turn aggro
to both content producers/distributors and end-customers.

Buuuuuuuuut no.. it was a silly short-lived comment about piracy (not even an
educating one)

~~~
inciampati
Actually it was a comment questioning why they don't use torrents to provide
their services. And also pointing out that p2p based solutions somehow
outperform theirs in many cases (for me, not US based so the net neutrality
stuff is maybe not relevant). Anyway sorry I was too silly and didn't clarify
these points.

~~~
Klathmon
P2P has traditionally not worked out well for companies.

Even recently Microsoft tried it for updates, and people were quite angry.

I honestly don't see a situation where someone would prefer to use their own
bandwidth for a paid service rather than having a server do the heavy lifting.
Either as the provider, or the consumer.

~~~
will4274
> people were quite angry.

I don't think this is true. Some enterprises were concerned about the security
aspects and asked for greater controls, which Microsoft provided, but Windows
10 has not removed P2P updates, AFAIK.

~~~
PuffinBlue
It has not but I think it's not enabled by default now, and you can declare
you want it but limit to a certain amount if on a metered connection, or just
turn it on fully.

I think it was the metered connection bit people were annoyed about,
especially those who were tethering to their phones and then it
downloaded/shared a huge amount of data.

~~~
will4274
> I think it's not enabled by default now

This is easy enough to check with a simple Google search. (It's still enabled
by default).

------
wz1000
> Netflix has more than 110 million subscribers. > Netflix operates in more
> than 200 countries

The five largest public torrent trackers track 30-50 million peers at any
given time, serving over 10 million torrents. Torrents are estimated to have
over a quarter of a billion active users. Torrents can be streamed, and in the
end you have a DRM free copy of the file that you can rewatch at your
pleasure, on any device(adding custom subtitles if you need them). It amazes
me how much effort has been expended on pushing malware to almost
everybody(binary, unauditable blobs in every browser) solely so that massive
cartels can make money on something that has been solved since 2001.

Subscribing to Netflix, or any other proprietary streaming (dis)service is not
only foolish(in the sense of surrendering your own freedom, not to mention
your hard earned money to nefarious corporate interests), but deeply
unethical.

~~~
madeofpalk
> not to mention your hard earned money to nefarious corporate interests

Yes, this is the basis of our society. I work for money to buy things, mostly
from corporations. There's nothing nefarious about that.

I find it really hard to believe that paying for content is unethical.

~~~
wz1000
> There's nothing nefarious about that.

[https://www.washingtonpost.com/news/the-
switch/wp/2013/10/25...](https://www.washingtonpost.com/news/the-
switch/wp/2013/10/25/15-years-ago-congress-kept-mickey-mouse-out-of-the-
public-domain-will-they-do-it-again/?utm_term=.018a8d20cb8d)

A small example of the nefarious behaviour you support by paying for Netflix.

> Yes, this is the basis of our society. I work for money to buy things,
> mostly from corporations

Intellectual Property is by definition "non-excludable", which means my having
access to some information does not exclude you from having access to the same
information, unlike land or food or clothing. It is very different from these,
and using your Government to violate the sovereignty of other nations and
individuals to impose your warped up notions of property is deeply unethical.

~~~
madeofpalk
What?

~~~
klapinat0r
I think he meant to use exclusivity as a straw man to show that _" IP is not
food"_, instead of the more often misused: _" not paying is not equal to
stealing"_.

Both are obviously false, but I think that was the intend (argument by
confusion?).

------
macpete
Feels more like an ad for AWS than a technical post

------
hetfeld
What happens when i press play?

It checks DRM settings in my browser and gets me lowest quality because my
browser is open source. That is what happens.

~~~
synotna
You have the freedom to watch something else

~~~
Walf
And content creators have the freedom to choose where they distribute content,
many of whom choose Netflix.

