
The Evolution of Container Usage at Netflix - kiyanwang
http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html
======
RcouF1uZ4gsC
Netflix seems so over engineered to me. They basically have a catalog of a few
thousand movies that are negotiated months in advance of actual use.
Basically, they just need to encode them and put them on a box and ship them
to edge caches. Caching immutable data scales incredibly well.I would also bet
that 99.99% of the movies people actually watch on Netflix, would fit on a
single box.

In regard to the analytics, they have 100 million subscribers. Let's say each
subscriber watches an average of 100 episodes/movies a day. For each watch you
record subscriber ID, movie ID, start time, stop time and get 32 bytes * 100 *
100,000,000 = 320 gigabytes of data per day total. I am pretty sure that you
could get a commercial database and and business intelligence package that
could support the type of analytics you need (mainly clustering analysis) at
that scale. A national grocery chain, probably has a similar amount of data
ingestion and a similar analytics need. In addition, I have subscribed to
multiple Netflix type services and I have never weighed the quality of
suggestions very high, giving much more weight to the functionality of the
client, lack of ads, and large catalog of good movies.

As evidence that this is a solved engineering problem, just look at the number
of similar movie services: Amazon, Google play, Hulu, PlayStation Vue,
Pureflix, Crackle, etc.

Google, Facebook, Baidu, Amazon, the self driving car companies are doing
cutting edge stuff in terms of scalability and analysis, but not Netflix. The
complexity of their operations, seems mainly to be one of their own doing and
not intrinsic to the service they provide.

So I look at stuff like the article here and see a bunch of very smart
engineers who are bored with the (solved) core problem and spend their time
making cool stuff which is actually a pretty good thing.

~~~
lmkg
At one point, Netflix video streaming accounted for something absurd like 20%
of all Internet traffic in the country. Regardless of whether it "scaled
incredibly well," I would imagine there are still novel issues with that much
scaling.

Reliability also matters differently for video than it does for normal web
traffic. It's one thing to shuttle 5GB (or whatever) of data over the course
of an hour. It's another thing to shuttle 5GB of data _with no hiccups_ for an
hour. Detecting and routing around machine or network issues fast enough that
real-time video playback is not impeded sounds to me like a difficult
challenge.

~~~
rsync
"Reliability also matters differently for video than it does for normal web
traffic. It's one thing to shuttle 5GB (or whatever) of data over the course
of an hour. It's another thing to shuttle 5GB of data with no hiccups for an
hour."

Speaking of which ...

I notice that while youtube continues to buffer video while paused, netflix
(and many, many other video players online) do not.

So while netflix will auto-adjust quality for you in response to a bad net
connection, you can't just pause it, go brush your teeth, and come back with a
big enough buffer to avoid skips through the entire video.

When did video players stop buffering-on-pause ? Why was that choice made ?

On the other end of the spectrum is whatever video player showtime online uses
- it does not adjust quality and it does not buffer on pause. Basically it was
built for perfect Internet connections and nothing else.

~~~
savanaly
Probably too many cases where they deliver a bunch of content while it's
paused and then the content never gets played. More than half the people who
pause it probably end up closing the tab without finishing the video, so one
can save a lot of bandwidth by only delivering the data when it's needed. A
totally unsupported hypothesis.

~~~
yomly
I arrived at the same conclusion when I realised that YouTube no longer
buffers the video all the way through to the end on mobile anymore. I miss
being able to buffer a (long) video for offline playback when riding public
transport.

~~~
joshbaptiste
Ditto, for this case I youtube-dl prior and watch on mobile VLC

------
hueving
>The theme that underlies all these improvements is developer innovation
velocity

I can't wait until this becomes buzzword de jour and startups start using it
in their product descriptions. Then someone needs to start talking about
products to "enable developer innovation acceleration" to outpace these crufty
companies stuck at 25 kph.

~~~
geodel
Yup. I think further down someone will 'democratize innovation velocity and
acceleration'

~~~
chii
"unlock innovation potential"

~~~
billfor
You have to do it holistically though.

~~~
RangerScience
Holistically-caused emergent innovation velocity and acceleration?

...The scary thing is I think that makes sense, actually. I can understand
that sentence as a thing I would want - Twiddle with your company culture so
that individuals come up with (and make) new ideas, in such a way that as time
goes on, their ability to do that grows...?

Makes sense to go at it from a systems view (holistic), rather than components
view.

How about -

Homeopathic synergies for creative empowerment and evolution?

That sounds properly almost, but not actually, sensical.

------
nailer
They're killing performance (one of the main reasons to use containers) and
adding a massive extra layer of management if they're running containers on
EC2.

I suspect Netflix are too wedded to AWS (which is weird as Amazon is their
biggest threat) but Triton or Red Shift (both of which actually isolate
containers using SmartOS and SELinux respectively) make way more sense for
other people who want to use the blazing fast IO speed of containers on bare
metal.

~~~
richardwhiuk
Or use something like GKE where the containers should be running directly on
hardware, so you only have one layer.

~~~
lima
Containers don't actually run directly on hardware with GCE - there's still a
virtualization layer in-between.

I'm 99% sure that Google runs one VM per container because that's the only way
to make it safe.

Anything else would be insane.

~~~
outworlder
> I'm 99% sure that Google runs one VM per container

I'm 100% sure you are wrong. You might as well just use VMs.

Containers are not only about safety, you know.

~~~
lima
I stand corrected, it's a bunch of VMs per customer but still no multi-
tenancy.

------
yeukhon
Netflix Engineering team amazes me a lot. They literally took all the
available apis, and build their own platform despite some of the features are
already in the AWS offerings. I suppose they did it mainly because the native
service isn't flexible and robust enough for their use cases.

BTW, their opening positions are always prefixed with "senior" title but I
guess that makes sense; Netflix builds pretty much everything from scratch
under time constraint.

~~~
tybit
They specifically only hire seniors, they don't want to provide positions for
juniors as they would prefer to pay more for more seniors than pay indirectly
for training juniors.

I think this reflects poorly on them, just as much as companies that use OSS
but don't contribute anything back.

~~~
JustSomeNobody
I guess "experience discrimination" is a thing now?

Seriously, if a company only wants to hire experienced folk, then so be it as
long as they don't exercise REAL discrimination.

------
sandGorgon
> _Today, we are in the process of rebuilding how we deploy device-specific
> server-side logic to our API tier leveraging single core optimized NodeJS
> servers_

Is this the core Netflix API ? have they moved from java - previously, their
entire open source contributions were around java
([https://netflix.github.io/](https://netflix.github.io/)). Hystrix repo was
updated barely a day ago.

For me, this is more interesting than the VM part.

~~~
aaronblohowiak
It is a splitting of the device-specific stuff from the general coordination
stuff. So the "core API" as you put it remains in java while letting the
device teams write their code in node (much of our UI teams are experts in JS
already).

Hystrix is important and won't be going away any time soon.

More info: [https://www.slideshare.net/mobile/KatharinaProbst/the-new-
ne...](https://www.slideshare.net/mobile/KatharinaProbst/the-new-netflix-api)

Disclaimer: I'm on paternity leave and not on those teams, but we've talked
publicly about this stuff recently..

~~~
sandGorgon
This is very interesting. And very indicative of convergence. If someone at
the same of Netflix has a strong drive towards cross pollination of talent
from device to server, then I suppose the js ecosystem is far more successful
than I thought.

------
dominotw
> We implemented multi-tenant isolation (CPU, memory, disk, networking and
> security) using a combination of Linux, Docker and our own isolation
> technology.

Curious what their 'own isolation technology' does that docker doesn't.

Also, what does Fenzo do that marathon doesn't . Looks like Fenzo sits on top
of marathon and sends it some sort of recommendations for scheduling. I need
to find a good example of what this is actually doing.

~~~
skrater
Justifying high engineering salaries, i see this a lot in great companies.

~~~
colanderman
Netflix is the last place I'd expect to do this. They're famous for letting
people go when they're no longer needed.

~~~
macNchz
That sounds like a great incentive to build big proprietary systems that
depend on your presence...

------
brianwawok
How does this compare to k8s? Seems like it has everything plus a layer of
scheduling and batch jobs on top?

~~~
scurvy
Doesn't kubernetes run an overlay network (at least in some scenarios)?

~~~
gtirloni
An overlay network is not a requirement. The only requirement is that pods
(collection of containers) should be able to communicate with each other
directly without NAT. Each pod gets an IP address in the container network.

Overlay network technologies (flannel, weave, calico, etc) are popular but
they aren't mandatory. You can implement it using hardware switches and VLANs
if you wish.

~~~
scurvy
Sorry, but what are you inferring by hardware switches? Something not an
overlay? Something that switches VXLAN in hardware?

------
RangerScience
> In each of these examples, a key to the success of Titus was deciding what
> Titus would not do, leveraging the full value other infrastructure teams
> provide.

This. So much this.

------
lkrubner
This is some amazing scale:

\-------------------

We run a peak of 500 r3.8xl instances in support of our batch users. That
represents 16,000 cores of compute with 120 TB of memory. We also added
support for GPUs as a resource type using p2.8xl instances to power deep
learning with neural nets and mini-batch.

In the early part of 2017, our stream-processing-as-a-service team decided to
leverage Titus to enable simpler and faster cluster management for their Flink
based system. This usage has resulted in over 10,000 service job containers
that are long running and re-deployed as stream processing jobs are changed.
These and other services use thousands of m4.4xl instances.

While the above use cases are critical to our business, issues with these
containers do not impact Netflix customers immediately. That has changed as
Titus containers recently started running services that satisfy Netflix
customer requests.

------
bjornlouser
"We run a peak of 500 r3.8xl instances in support of our batch users. That
represents 16,000 cores of compute with 120 TB of memory."

I get that they are bragging about their implementation, but what number for
peak batch processing instances would they be embarrassed to divulge?

------
johnsmith21006
What has happened to HN? This thread is so filled with misinformation it is
clear there is very little understanding of container.

A container is just a process. Really no different than any other. So cache
and shared libraries, etc all the same with just a little care.

------
rodionos
I'm looking at Netflix repos at
[https://github.com/Netflix](https://github.com/Netflix). 5 pages x 30, that's
a lot of repos.

------
gbrown_
Is anyone else having issues accessing this?

    
    
        $ curl http://techblog.netflix.com/2017/04/the-evolution-of-container-usage-at.html
        <!DOCTYPE html>
        <html><head>
        <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
        <title>Access Denied</title>
        <style type="text/css">body {margin:0;font-family:verdana,sans-serif;} h1 {margin:0;padding:12px 25px;background-color:#343434;color:#ddd} p {margin:12px 25px;} strong {color:#E0042D;}</style>
        </head>
        <body>
        <h1>Access Denied</h1>
        <p>
        <strong>You are attempting to access a forbidden site.</strong><br/><br/>
        Consult your system administrator for details.
        </p>
        </body>

~~~
thealfreds
Work filtering out netflix.com? Old place use to do it and it would bug me I
couldn't read their techblog at work.

~~~
gbrown_
Yup that appears to be it. Didn't even realise my workplace filtered anything.

------
bertlequant
I would've imagined Netflix had their own hardware, both compute and storage.

~~~
nailer
I've heard they're looking at 'architecture in a box' solutions, which would
allow them to abstract designs from specific cloud providers and perform cloud
arbitrage; eg, if Cloud Provider X can run the app at 14c/hour, and Cloud
Provider Y can run the app at 22c/hour, then they can just deploy the entire
architecture on Cloud Provider Y.

~~~
jjnoakes
Why would they choose the more expensive one?

~~~
nailer
Because, er, I made a typo and it's too late to edit. Good spot!

------
wahnfrieden
I wonder how this compares with the recently launched AWS Batch.

