
A View of the Entire Netflix Stack - hepha1979
http://highscalability.com/blog/2015/11/9/a-360-degree-view-of-the-entire-netflix-stack.html
======
morgante
@monsky: (Somehow this got disconnected from the thread below.)

The headline of the slide you pulled that from is literally "Hard Work—Not
Relevant."

Netflix doesn't care if you work 100 hours a week, they care how well you
perform. If you can do an amazing job in 30 hours a week, they're one of the
few places which would actually embrace you.

> I'm not sure how Netflix grades things... but in school B is "Above
> average".

So what? Why on Earth should Netflix be targeting "average" workers when they
deliberately pay twice the average.

That's about as ridiculous as saying that Harvard should stop admitting A
students because "B" is average.

Netflix has a deliberate strategy of keeping their engineering bar very high
and paying accordingly. That doesn't mean they want you to sacrifice your life
to the company.

Also, Netflix is in no way a startup.

~~~
erikb
The problem about the idea of "doing great things in 30 hours" is that people
who are not directly involved with implementation and deployment usually think
that each and every 2 day task takes at most 2 hours, because we humans are
unable to expect the complexity that most tasks have on this planet. (every
time you think about a college, super market employee etc "just do your job
man" you are experiencing that fallacy)

And then you as the 30 hours employee are faced with not delivering 40 hours
of work in 30 hours (which is possible if you are smarter, more involved, have
more experience etc) but no you are expected to do 160 hours of work in 30
hours, because you don't compete with the other engineers but you compete with
the expectation of your manager.

We need to accept that most people (colleagues, employees, customers, bosses)
will always be disappointed by the results we achieved because they had
unrealistic expectations to begin with. And we must excuse them because we do
the same the other way around. That is why "40 hour week" is so relevant.
Because then you can say after working hard for 40 hours, really doing all you
can, maybe even achieving 30% more than your colleagues, that time is up and
you need to take care of your children, wife, parents, friends and hobbies you
may have.

Now let's assume for a moment that all I said was BS. Let's assume I'm super
untalented and for most people it's actually possible to achieve in 30 hours
enough to impress their leaders. Now you do it once. You do it twice. Maybe
thrice. But at that point your leader will adjust in his head, maybe without
realising, what 100% means for you. So the fourth time he doesn't think it's
awesome anymore. The tenth time he thinks you are lazy. And now you have to
achieve 50 hours of work in 30 hours. It still doesn't work out.

~~~
smokeyj
> And now you have to achieve 50 hours of work in 30 hours.

Could this argument not be made about the 40hr week?

~~~
amouat
Only if everybody else was working 50 hour weeks.

~~~
smokeyj
It has nothing to do with everyone else. If you impressed your boss while
sticking to a 40hr week, they'll only be impressed so long before it becomes
your norm. In which case you bump up your hours to maintain your prestige. I
don't buy that reasoning, just saying it goes both ways.

------
monksy
> Sustained B-level performance, despite "A for effort", generates a generous
> severance package, with respect.

You've got to love these startups who basically say "if you're not giving us
everything beyond what we expect... go screw yourself. I'm not sure how
Netflix grades things... but in school B is "Above average".

Is Netflix looking to employee and develop people or are they looking to
extract life from people?

~~~
hitekker
Probably the latter. In my view, Netflix (IBM, Amazon, Bloomberg etc) should
consider what would happen if they were completely honest to their employees:

"You are a cog in our machine. We do not care for you. We only care for your
work. We will pay you insofar as you can execute on our vision. Further, you
will be paid according to how hard it is to replace you. If you can be
replaced by someone who executes significantly better, relative to your pay,
we will tell you to leave."

Needless to say, stating that would make many of their employees, especially
the weak-minded and insecure (those who defend their terrible culture),
unhappy. I imagine even those in charge of the culture, speaking those words,
would feel bad.

And they should feel bad. While such overwhelmingly negative words should
almost never be spoken with authority, these decision makers must at least be
able to answer the question "What are we _really_ saying to our people?"
truthfully. And if the answer is something along the lines of "you're a cog in
the machine", then something has gone very wrong[1]

In my non-expert opinion, if a company truly wishes to be a "good" place to
work, it should know the distance between their words and the truth, and then
focus on minimizing that distance.

Spoken more simply: the more dishonest a company is to its employees, the
worse the culture.

[1] Or right, if the company requires joylessness to turn a profit.

~~~
pessimizer
>"You are a cog in our machine. We do not care for you. We only care for your
work. We will pay you insofar as you can execute on our vision. Further, you
will be paid according to how hard it is to replace you. If you can be
replaced by someone who executes significantly better, relative to your pay,
we will tell you to leave."

Isn't this the ideal of wage labor and the reality of 98% of jobs? I've never
even suspected that my corporate employer cares about me as a person aside
from how I produce, and have never suspected that I'm being paid by them for
any other reason that it would cost them a lot more than they pay me to find
somebody to replace me. I don't mean to say I haven't had great friendships
and working relationships with management, but to the business, I'm a line
item. When I notice a business making decisions contrary to the bottom line, I
suspect corruption, not corporate affection.

They're a cog in my machine, too. I can only depend on that cog if it behaves
rationally.

~~~
hitekker
I agree it's the reality for a majority if not the super majority of jobs. I
also think it could be the ideal of hourly wage labor: which is to say that
the company wants nothing more than what they can motivate you to do with
money, and that you want nothing more from the company than their money.

Looking back, I see I made my comment in the context of Silicon Valley
behemoths and startups: companies that can both pretend to, and potentially
change some fundamental way of doing things.

I think the difference in my mind between "creative" companies and normal ones
is the "potential", achievements outside of a strictly defined expectations of
contract, for both the business and the employee. Maybe a great insight, a new
tool... anything that greatly benefits both the business and the employee,
especially if the employee uses the product(s) and believes in the mission.

When you take away that potential (by saying employees are just cogs in a
machine), you might take with it the creative spark that these hopeful
companies ostensibly rely on. In my mind, that "potential" is the real joy of
a job, which is my reason for the (slightly snarky) comment that "cog in the
machine" is only acceptable "if the company requires joylessness to turn a
profit". If I could, I would edit that post to be more clear/to take into
account businesses which do not require any specialness.

At least for my central point, "the more dishonest, the worst the culture"...
I believe that stands, especially in circumstances where a person is (somehow)
manufacturing toilet paper and their supervisor is telling them they are
(somehow) on their way to curing cancer.

~~~
pessimizer
> If I could, I would edit that post to be more clear/to take into account
> businesses which do not require any specialness.

That I'm creative (amongst other things) is why they pay me well. My
successful experiments have a record of adding value well above my pay even if
aggregated with my many unsuccessful experiments, although I try my best not
to keep it that way for long. I'm not sure I've had any technical job with
completely "strictly defined expectations of contract", I think that's a red
herring. It sounds nice that the most creative and innovative
projects/companies give less of a shit about the bottom line; I personally
think that the bottom line is the fuel for creativity (didn't have much money
when I was younger.) Anybody can sell your product with unlimited money to
spill, but doing it with the best profit margin possible is creativity.

I like to think I'm a special snowflake...

------
yarou
This reads less like a technical overview of Netflix's stack and more like an
advertisement for working there.

I'll give you my two cents on the matter. Netflix spins a great narrative and
ideal -- the rose-colored "meritocratic" dream of Silicon Valley.

I'd suspect the reality of working there is a lot more mundane than the
beautiful narrative that they (and by extension, the author of the post) are
trying to sell you.

There are probably a wide array of experiences that current employees have,
ranging from pure joy to misery, which are contingent upon team, co-workers,
management, compensation, etc.

An interesting read nonetheless.

------
faizshah
Adrian Cockcroft gave a much better overview of their open source projects
here:
[https://www.youtube.com/watch?v=R2kKmMyqTfc](https://www.youtube.com/watch?v=R2kKmMyqTfc)

It's two hours long but the first hour is sufficient to get an overview of
their open source projects.

~~~
mxpxrocks10
Good call. Thanks for posting. We'll add this to the comments of the original
article.

~~~
BinaryRage
You might want to link to the OSS site too -
[https://netflix.github.io/](https://netflix.github.io/)

------
clarle
Only the UI layer of Netflix is in Node.

Most of the cloud services are JVM-based (as is much of the NetflixOSS
ecosystem [1]), and then the Node services connect to the JVM-based services
via Prana. [2]

[1]: [http://netflix.github.io/](http://netflix.github.io/)

[2]: [http://ispyker.blogspot.com/2015/10/towards-being-better-
abo...](http://ispyker.blogspot.com/2015/10/towards-being-better-about-open-
source.html)

------
sciurus
This isn't a very good article. Judging from the first table, netflix is
written in node.js using python and java. :-/

~~~
mxpxrocks10
Thanks for the feedback. We heard this a little bit through twitter. Can you
reference any links? Happy to make the change. It was a little tricky to put
together with out interviewing someone. We want it to be accurate and useful.
Please post any reference links here and we'll follow up with an edit.

~~~
andrew_mason1
Well, node.js implies that at least part of Netflix is written in JavaScript,
yet the only listed languages are Python and Java.

~~~
zappo2938
Someone at Netflix did a talk perhaps it was about Falcor. They use React
embedded in browser apps across all devices. So they use Javascript on the
front end. They use Node.js as the proxy server / router with a framework
called Falcor which is written in Javascript. Falcor queries all the
multitudes of services and databases then caches that data in a graph. Falcor
reaches out to all the servers and databases, scores or services, that are
written in Python and Java.

~~~
mikeryan
Picking a nit, they don't use React in the browser across all devices. They
actually use an internal rendering engine called Gibbon on some devices with
client code written JavaScript with React but not rendering to HTML or a
browser instead using their internal engine. Interestingly this has become
easier with React breaking out React and ReactDOM.

Interesting to note that Netflix must see Gibbon as a competitive advantage,
I've been told its not likely to see an open source release.

[https://www.youtube.com/watch?v=eNC0mRYGWgc](https://www.youtube.com/watch?v=eNC0mRYGWgc)

------
fitzwatermellow
This AWS re:Invent 2015 talk by Dave Hahn is also really illuminating: A Day
in the Life of a Netflix Engineer (Using 37% of the Internet)

[https://www.youtube.com/watch?v=-mL3zT1iIKw](https://www.youtube.com/watch?v=-mL3zT1iIKw)

Interesting to me that they use Atlassian Confluence for team stuff. Thinking
of testing it out as well...

~~~
swsieber
Confluence is pretty cool, but they suck at bullet lists and indenting code
blocks... so basically they suck at a couple of basic UI stuff. Apart from
that it seems pretty great.

~~~
georgebarnett
Are you aware of the {code} macro which formats code?

~~~
swsieber
I am. It does not indent properly though - there is only one level of stable
code indentation (the entire block that is; it is possible to indent the code
within the block, but I'd like to indent the code frame).

------
zobzu
The A (promoted) vs B grade (fired) perf means:

\- No "regularly well performing" employees are kept. You're either promoted
or fired. Ensure workforce moves.

\- Doing the shitty necessary work such as fixing tech-debt just gets you
fired. But I bet people do it anyway, else Netflix would be in troubles. It's
just that they get fired after that. Not really nice.

It also means you've to be only doing stuff that has direct visible impact,
and refuse any work that does not/place yourself visually above others.

Not saying this is a bad strategy to get stuff to work, not sure if very
ethical though.

That's all I noticed was actually interesting ;-)

~~~
kyllo
Exactly--in a work environment when everyone is fearing for their livelihood,
succeeding at performance theatre becomes the primary concern. You need to be
a top _performer_ , in both senses of the word--but it's human nature that
over time the acting sense will take priority.

------
kylek
Not sure if it was left out on purpose, but I'd love to see more details on
their transcoding stack (they only mention that it is done in EC2)

~~~
mxpxrocks10
Hi Kylek - thanks for the feedback. It was a little hard on first try to find
information on this. The only thing that seemed clear was that adaptive bit
rate was done on the edge and transcoding was done and stored. Feel free to
post links here if anyone has any.

------
lappa
This is a bit confusing. I don't suspect Netflix is built using primarily
node.js, and I don't see how the Netflix Prize (algorithm for predicting what
a person is most likely to want to watch) falls under "scaling".

~~~
johansch
This is nitpicking.

~~~
such_a_casual
Honestly, there are enough problems with the article to warrant a critique.
However, someone may not want to take the time to outline every single one.
It's incredibly confusing to the reader that the article states Netflix relies
on frameworks for one language, and then states that all of their code is
written in another language.

~~~
mxpxrocks10
Hey, we got this fixed up on the original post- waiting for Todd to fix the
repost on HighScalability shortly. If there are some more problems let us
know. We want it to be accurate. [http://www.scalescale.com/the-stack-behind-
netflix-scaling/](http://www.scalescale.com/the-stack-behind-netflix-scaling/)

------
CodeSheikh
I am a bit surprised. Why host Netflix application on Amazon EC2 and not have
your own data centers? A startup hosting at Amazon EC2 makes sense

~~~
rpedela
Why Netflix chose AWS: [http://techblog.netflix.com/2010/12/four-reasons-we-
choose-a...](http://techblog.netflix.com/2010/12/four-reasons-we-choose-
amazons-cloud-as.html)

Also Quora answer: [https://www.quora.com/Why-does-Netflix-use-Amazon-
EC2](https://www.quora.com/Why-does-Netflix-use-Amazon-EC2)

~~~
mentos
The reason that strikes me as most important is #2 "Letting Amazon focus on
data center infrastructure allows our engineers to focus on building and
improving our business."

Too many companies roll their own solutions when they could take advantage of
economies of scale like this and let another company dominate the problem.

Obviously there are privacy reasons to own your own data but I don't think
they outweigh the performance and efficiency implications.

~~~
qaq
aws and performance :) Crappy performance and poor uptime is precisely why
people move off aws.

~~~
jadavis
So, um, why does Netflix always seem to be up/performant?

~~~
njharman
Because they actually know how to implement fault tolerant scale on AWS. Most
companies don't have simian army randomly wrecking infrastructure. Many
companies/engineers think they can spin up a few (or few hundred) ec2
instances and think they have solved resiliency.

AWS is a tool, not a bullet made of silver.

------
orestes910
I really like that Netflix is constantly open about how their systems are run.
In terms of raw throughput not a lot of companies can hold a candle to them,
so its cool to see how the big players do things.

~~~
XorNot
I would wager because, looking at that technology stack, most of it is
commercial paid services? "We use EC2" just pushes the question to someone
else.

------
JupiterMoon
So which part of this stack is to blame for their slow clunky website? (NB
their website used to work well enough on my devices the newer one is the
problem.)

------
rufugee
Over the years, I've heard a few podcasts stating that Netflix relies heavily
on Grails/Groovy for infrastructure management tools
([https://github.com/Netflix/asgard](https://github.com/Netflix/asgard), etc).
Wonder why that isn't mentioned?

~~~
vorg
They do say: "We are sharing the things we think are most interesting." Maybe
a minor use of Groovy and Grails is only interesting to people trying to
promote those products.

------
johansch
Is this reliance on a variety of external services why Netflix fails so often
compared to e.g. youtube.com? I assume so.

~~~
mxpxrocks10
[http://mashable.com/2015/11/10/netflix-site-goes-
down/](http://mashable.com/2015/11/10/netflix-site-goes-down/)

------
insulanian
Where is the Chaos Monkey?

([http://techblog.netflix.com/2012/07/chaos-monkey-released-
in...](http://techblog.netflix.com/2012/07/chaos-monkey-released-into-
wild.html))

~~~
mxpxrocks10
Simian Army Suite
[https://github.com/Netflix/SimianArmy](https://github.com/Netflix/SimianArmy)

------
steven2012
As the comment at the end of the article mentioned, they forgot to add Spring.
I've met a few Netflix engineers, and they said their Java implementation is
done using Spring.

~~~
mxpxrocks10
hey Steven! Thanks. Do you have any links that reference this? We want it to
be as accurate as possible.

~~~
steven2012
No links, I got this from actually talking with some engineers. Have you not
contacted a single Netflix engineer to verify your article? If not, I find
that bizarre.

------
mnml_
It must be really expensive tho (even with discounts). But I guess they don't
care.

------
Randgalt
This view is missing some things. For example, Curator (now at Apache).

~~~
mxpxrocks10
[http://curator.apache.org/](http://curator.apache.org/) "Guava is to Java
what Curator is to ZooKeeper" Patrick Hunt, ZooKeeper committer

------
brown9-2
Source for the bit on Mesos?

~~~
mxpxrocks10
check: [http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-
for-...](http://techblog.netflix.com/2015/08/fenzo-oss-scheduler-for-apache-
mesos.html) and post anything else

------
such_a_casual
A word to the author: Please proof read your writing.

------
bricss
OMG! Node.js in frameworks folks xD

------
chatmasta
Funny timing given that Netflix is currently down... [0]

[0]
[http://www.isitdownrightnow.com/netflix.com.html](http://www.isitdownrightnow.com/netflix.com.html)

------
Geee
Why there's so much brag about Netflix while they can't fix even simple bugs
and UX issues? After all, it's one of the simplest apps there are. Whole stack
could be handled by one guy.

~~~
chambo622
Curious what you mean by this, as it's overall one of the more polished
products I use on a regular basis. Hasn't always been the case, and the
Android app still does some weird UI things, but generally the product is
solid and a pleasure to use.

"Whole stack could be handled by one guy" \- comments like like this and
similar others ("I could build Twitter in a weekend", etc.) strike me as
unproductive.

~~~
Geee
For example, Netflix consistently forgets what I have seen and if I don't
watch the ending credits it thinks I haven't watched an item and suggests it
in the continue watching section. There should be visual indicator on what I
have watched, and if there are unwatched episodes in series. Now I have to
open it and go to the episode list, which is unreliable. Also, starting video
playback is too slow. It's not possible to open items in new tabs. There are
no keyboard controls.

