

Linux Profiling at Netflix - 2510c39011c5
http://www.slideshare.net/brendangregg/scale2015-linux-perfprofiling

======
rogerbinns
Can anyone speak as to why Netflix needs so much computing power? As far as I
can tell, content delivery is handled separately, so all these talks are about
the non-content side.

Sure we need to see various personalised lists when first connecting, but no
activity while actually watching things. The personalisation seems very light
- eg their lists of comedies look like they show the same to everyone. And it
doesn't seem intelligent at all, like for example hiding items you have
already seen.
[https://news.ycombinator.com/item?id=9083196](https://news.ycombinator.com/item?id=9083196)

~~~
cpwatson
The microservice architecture results in a larger footprint than you might
find with a more monolithic app stack; which provides us with improved
engineering velocity and reliability. Although the content playback is served
by our CDN, all other device interactions are handled by the services on AWS.
We also have some blogposts which provide a glimpse into some of the work we
do behind the scenes ([http://techblog.netflix.com/2015/01/netflixs-viewing-
data-ho...](http://techblog.netflix.com/2015/01/netflixs-viewing-data-how-we-
know-where.html)). Oh yeah, we also encode and run our big data services on
AWS...

~~~
rogerbinns
This still doesn't answer why it is so large. Or more accurately my experience
as a Netflix customer doesn't reflect the apparent size of all these backends.
Compare to something like Reddit where every page is customised, I go through
lots of them, and my interactions are frequent. My Netflix interactions look
like recently viewed being custom, and everything else being generic (same for
all users). If you told me my recommendations are recalculated once a month
I'd believe you. The majority of time is spent viewing which is just periodic
progress updates. In other words my user experience suggests Reddit having an
infrastructure many times larger than Netflix.

~~~
mryan
I read that Netflix encodes each video 100+ times to ensure good performance
across a wide range of devices. In addition to the "social network" side of
the site (recommendations, what your friends watch, etc.) there is a lot of
processing going on in the background.

~~~
rogerbinns
I vaguely recall the number being encoded 120 different ways. In any event
this is independent of the number of users, and proportional to the amount of
new incoming content. It also isn't real time, and doesn't really matter how
long it takes other than marketing deadlines when they announce a new season
of something far in advance.

The recommendations don't show any evidence of being calculated frequently,
nor do they have to be done in real time. There is certainly _way_ less
appearing to go on than a site like Reddit.

"There is a lot of processing going on in the background" doesn't really
answer the question :-)

~~~
mryan
Sure, this processing does not need to happen in real time, but it _does_ need
to happen.

My point was that to assume the user-visible portion of the site accounts for
most of their computing needs is probably incorrect. Video encoding was just
one of the first things I thought of, and I bet there are many more.

I admit "processing going on in the background" was a real hand-wavey way to
phrase it, but I think you know what I mean. It's an iceberg - there's a lot
happening under the surface. :-)

------
planckscnst
The SCALE 13x videos are going up here:
[https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ](https://www.youtube.com/channel/UCN2nbMPLJWv3Y__4JuF_hMQ)
Does anyone know in what room and day Brendan spoke?

~~~
rollardurby
[https://www.youtube.com/watch?v=dhDpncTDDvI#t=892](https://www.youtube.com/watch?v=dhDpncTDDvI#t=892)

~~~
teddyh
You can format the time like this:

[https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s](https://www.youtube.com/watch?v=dhDpncTDDvI#t=14m52s)

------
deirdres
For those who missed today's post
[https://news.ycombinator.com/item?id=9121015](https://news.ycombinator.com/item?id=9121015)
because it got downvoted (wrongly) for being a duplicate: accompanying blog
post and video now available:
[http://www.brendangregg.com/blog/2015-02-27/linux-
profiling-...](http://www.brendangregg.com/blog/2015-02-27/linux-profiling-at-
netflix.html)

------
ctb_mg
I really like the workflow chart on slide 22. I can't help but think this is a
useful way to visualize common workflow sequences of commands (think git add,
git commit, git push).

Honestly I wouldn't mind if man pages came with ascii art to visualize this
type of documentation.

------
krakensden
It often feels like the only users of perf are its developers and Brendan
Gregg.

~~~
scott_s
I use it - perf is wonderful. Back in the dark days of grad school, getting
performance counter data for your program required hacking your code to call
into PAPI
([http://icl.cs.utk.edu/papi/overview/](http://icl.cs.utk.edu/papi/overview/)).
Being able to tell a program, "Hey, run this for me, and tell me how many
branch mispredictions there are. And where they happen" is like _magic_.

If you do any performance analysis of software, learn perf.

Official manual:
[https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page)

Brendan Gregg's perf page:
[http://www.brendangregg.com/perf.html](http://www.brendangregg.com/perf.html)

------
X-Istence
This talk was absolutely fantastic at SCALE13x. Lot of good content that will
help a lot of people get started with CPU profiling.

------
distantsounds
Those flame graphs look hella useful, hopefully the JDK patches make their way
in so we don't have to futz around with customizing too much. Thanks for this!

~~~
MrBuddyCasino
Absolutely, I had no idea there is a way to export the symbols and make them
known to perf. Its magic, and I want it.

------
nwmcsween
Omitting the frame pointer is a default on x86_64 its specified in the abi so
it's going to be mighty tough to change that. I don't understand why you
recommend not using the debug info when you already have to use it for symbol
name resolution.

~~~
brendangregg
Are you talking about
[http://www.x86-64.org/documentation/abi.pdf](http://www.x86-64.org/documentation/abi.pdf),
and footnote 7 on page 16, which reads:

"7 The conventional use of %rbp as a frame pointer for the stack frame may be
avoided by using %rsp (the stack pointer) to index into the stack frame. This
technique saves two instructions in the prologue and epilogue and makes one
additional general-purpose register (%rbp) available."

%rbp is mentioned elsewhere, eg, "optionally used as frame pointer"

This ABI does not appear to specify that the frame pointer is omitted. It uses
the words "may" and "optionally". Which means, the gcc engineers can choose to
do what they want. And they should choose to keep the frame pointer _by
default_, and _optionally_ omit it (for when the sacrifice is deemed worth
it): -fomit-frame-pointer. This would better follow the language used in the
ABI! :-)

We've found debuginfo to be large (size, bytes), which can slow down instance
creation, which means we must run with a higher headroom to accommodate. A
stripped-down debuginfo that just enabled stack walking might do.

~~~
nwmcsween
Compress the debug info? Gdb handles it no idea about perf though, there is
also dwo to remove redundant debug info.

------
api
Why FreeBSD for content delivery?

~~~
emaste
From
[https://openconnect.itp.netflix.com/software/index.html](https://openconnect.itp.netflix.com/software/index.html):

For the operating system, we use FreeBSD version 10. This was selected for its
balance of stability and features, a strong development community and staff
expertise. All code improvements, feature additions, and bug fixes are
contributed directly back to the open source community via the FreeBSD
committers on our team. We also strive to stay at the front of the FreeBSD
development process, allowing us to have a tight feedback loop with other
community and partner developers. The result has been a positive open source
ecosystem that lowers our development costs and multiplies the effectiveness
of our efforts..

------
icantthinkofone
So some aren't confused. Content delivery is on FreeBSD, not Linux.

~~~
ThreeFinger
ZFS drives?

------
shmerl
Surely they use Linux, yet they still can't offer a proper Linux client. DRMed
much.

