
Netflix FlameScope - brendangregg
https://medium.com/@NetflixTechBlog/netflix-flamescope-a57ca19d47bb
======
throwaway84742
I’ve done something like this at Microsoft in 2002. Except in my case it was a
multi-user system and each row represented a logical user, or a range of users
in a range partitioned space. You could vary granularity, click through to see
what’s busted, zoom into user ranges, discover correlated failures, discover
cases when something is failing for just one user consistently, correlate all
of it to performance counters, etc, etc. No one gave a shit.

~~~
mistermann
What did you use to build it?

~~~
throwaway84742
IIRC the standard MS stack at the time: C#, ASP.NET (which had just come out),
MS SQL, IE, some in memory caching and bitmap rendering (canvas didn’t exist),
AJAX. Pretty bleeding edge, but it all worked OK.

~~~
chrisweekly
Given when you were using that then-bleeding-edge stack, I wonder if you used
XML (ie, the "X" in the then-acronym AJAX _).

_ later de-acronym-ized to "Ajax" w/ JSON replacing XML.

~~~
throwaway84742
Of course. JSON wasn’t popularized back then. But mostly it was about
reloading parts of the page through DOM manipulation.

~~~
chrisweekly
Yeah, I was there too. (Web dev for a living since 1998.) I remember sending
HTML fragments over the wire too...

~~~
ams6110
If data were in XML format, XSLT could be used from javascript (or vbscript)
in the browsr for the DOM generation. Was pretty slick up to a point, but
since it was a blocking operation anything beyond a certain size started to
noticeably lock up the browser.

~~~
throwaway84742
I’ve done some of that as well, in different contexts. Those were the days.
Virgin territory, uncomplicated, no cross-browser concerns. I haven’t touched
frontend since 2005, and looking at it now it’s horrifying. Even the hairiest
C++ backend is easier for me now, just because once you write it it doesn’t
need to be “updated” for the next decade. Frontend seems to change every 6
months for no apparent benefit.

~~~
chrisweekly
> "no cross-browser concerns"

wat.

the original browser wars? ns vs ie? 2 incompatible implementations w/ no
tooling to help?

~~~
throwaway84742
It was a Microsoft tool for Windows only. There wasn’t even a question that
other browsers should not be supported.

------
cortesi
This is lovely. An interesting thing to think about here is the layout of the
sequential time-series data. This visualisation breaks the sequence out into
columns, which means that squares that are close to each other in the sequence
don't cluster visually.

I've played with using space-filling curves for this to visualise binaries:

[https://binvis.io/](https://binvis.io/)

You can play around on there with the scan layout (similar to the Netflix
layout) and compare it to the Hilbert curve. You can click and drag to select
regions in both (similar to the video for this visualisation).

There are tradeoffs - the Hilbert layout is not intuitive, and you have to
play around to select regions accurately. On the other hand, the clustering
lets you visually pick hotspots in much greater detail, e.g.:

[https://binvis.io/#/view/examples/tcpview.exe.bin?colors=ent...](https://binvis.io/#/view/examples/tcpview.exe.bin?colors=entropy)

I've found it useful to be able to toggle between both scanning and clustered
layouts, and plan to explore some more interaction paradigms when I get time
to turn back to the project.

~~~
brendangregg
I did binary visualizations many years ago too:

[https://github.com/brendangregg/Dump2PNG/wiki](https://github.com/brendangregg/Dump2PNG/wiki)

So with the subsecond-offset heat map, you get clustering from one column to
the next for events that happen at the same time offset. Which is common with
periodic tasks: see the example patterns in the post.

~~~
cortesi
It looks like we've traveled a similar path. The precursor to binvis is
scurve, a (slow, dodgy) command-line tool to do basically what dump2png does:

[https://github.com/cortesi/scurve](https://github.com/cortesi/scurve)

I have a blazing fast command-line version of binvis in Rust that I plan to
release in the next month or so. I really need make some time to turn back to
this.

~~~
lakechfoma
Wow I haven't seen your name in years, I had forgotten! I just want to thank
you for having an incredible blog and visualization tools. sortvis.org was
frequently a resource I used to learn sorting and then later as a tutor I
shared it with other students to help them learn. I've shared sortvis and the
posts you did on hilbert curves with young curious family members to inspire
them, and it always did! So thank you!

And Brendans blog has been more recently inspiring and educational, seeing you
guys chatting on HN is too cool. Thank you both for the awesome and freely
accessible material.

~~~
cortesi
My pleasure! Sortvis is down at the moment, but will be back up soon. I've
been swallowed up in other projects, but I have a perpetual new year's
resolution to write more blog posts. Keep an eye out in the next few months.
:)

------
smaili
Link to the repo --
[https://github.com/Netflix/flamescope](https://github.com/Netflix/flamescope)

From a quick glance, it's interesting to see both Flask and Node.JS be used
within the same project repo. Is this becoming more commonplace?

~~~
mspier
Not Node.JS specifically, but just npm for dependencies and task running to
build the React client from source. If you're not looking to build the client
from source, the distributed client bundle should suffice and only Python is
required.

------
itissid
> a) could not reliably capture a one minute flame graph covering the issue,
> as the onset of it kept fluctuating; b) capturing a two or three minute
> flamegraph didn’t help either, and the issue was “drowned” in the normal
> workload profile. Vadim asked me for help.

One interesting thing here is also the statistical nature of this problem:
i.e. there are spikes but their duration are not exact or known when to start.
Here[1] is some reference work done on Neural Spike Trains that seems
relevant.

[1][http://www.stat.columbia.edu/~liam/teaching/neurostat-
spr12/...](http://www.stat.columbia.edu/~liam/teaching/neurostat-
spr12/papers/brown-et-al/time-rescaling.pdf)

------
wkz
Wow, great visualization! Thanks for sharing the source.

A minor suggestion: For the first part of the demo i was thrown of by the fact
that each second as laid out bottom->top. Maybe moving the time labels below
the plot would provide a visual cue for the direction.

~~~
lscotte
I think this is intentional, as the norm is for 0 on a Y-axis to be at the
bottom and increase going up.

I can understand why it's confusing, and perhaps could be a UI option to
reverse it, but I do think that's the reason.

~~~
brendangregg
Yes, that's the reason! It means we get a t=0 origin for both the x- and
y-axis. But yes, we can also add UI options to reverse it and other things.

------
jsmeaton
Does anyone have any good tools or processes for generating the kind of
performance counters and information needed by this tool with a running python
application?

I'd love to be able to generate a flamegraph for a django app, for example,
and visualise it with flame scope.

[https://mail.python.org/pipermail/python-
dev/2014-November/1...](https://mail.python.org/pipermail/python-
dev/2014-November/136981.html) discusses the problems with python and the
`perf` tool. There may have been some patches applied recently to help, but it
doesn't look like there was a full resolution.

~~~
gnufx
Not a direct answer, but there's Python support of some sort available for at
least TAU
<[https://www.cs.uoregon.edu/research/tau/home.php>](https://www.cs.uoregon.edu/research/tau/home.php>),
Extrae <[https://tools.bsc.es/extrae>](https://tools.bsc.es/extrae>), and
Score-P
<[https://github.com/score-p/scorep_binding_python>](https://github.com/score-p/scorep_binding_python>).

------
zawerf
What's the purpose of visualizing it as grid? I thought it might be good for
debugging periodic events (e.g., every 16ms for rendering at 60fps) but it
seems like this is fixed to one sec for one column.

For example in the flame graph in chrome's performance tab, you can
select/slice/zoom/pan arbitrary time ranges. If you want the high level view
you just zoom out.

~~~
brendangregg
Supporting different y-axis ranges is something we can add, and I was already
thinking of adding a 1ms range for visualizing low-level CPU cycle activities
(which span nano and microseconds). I like the idea of 16ms for 60fps
analysis, where every column is one frame. Could help game development. :)

~~~
saulrh
Actually, seeing the diagonal stripes in the examples [1], maybe it'd be
useful to have a mode that lets you rescale the axes to line the stripes up?
If you're using this to inspect patterns, it seems to me that the most useful
time scale would not be a nice round human number, it'd be one that matches
well with the patterns you're trying to inspect - one period, not one second.

Way out in the distance, maybe even a variable-interval mode, where you can
click-select blocks and each one is given its own column, so you can directly
compare them without the dead space in the middle and starting from the same
baseline?

[1] Particularly striking: [https://cdn-
images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJu...](https://cdn-
images-1.medium.com/max/2000/1*EypUnkPtayeKr9bJuvgNEg.png)

~~~
Scaevolus
You could autoscale the y axis to whatever makes the columns have the most
similarity (minimize squared differences).

~~~
simsla
Rather than lsq, you could use autocorrelation within rows to line them up.

~~~
rakoo
This thing is extremely periodic. You should do an FFT, there should be a peak
that's more obvious than the others. That's your frequency and thus your
period.

------
wpietri
Nice. I had some search cluster optimization problems that this would have
been very handy for. I laughed when I got to "Ah, that’s Java garbage
collection," as that turned out to be one of the big contributors to variance
in response time.

------
kapitalx
Longer video with more examples:
[https://www.youtube.com/watch?v=gRawd7CO-Q8](https://www.youtube.com/watch?v=gRawd7CO-Q8)

------
gregdunn
Awesome!

It'd be pretty sweet if you could send things sampled using Vector over to
FlameScope. I know the flamegraphs PMDA is experimental but it's always worked
fine for me :)

(It'd also be pretty sweet if there was a fleshed out PMDA for bcc tools. And
Vector supported them. And, and, and... I have a really long wishlist.)

~~~
brendangregg
We'll get it done (although I'm not exactly sure who "we" will end up being
yet!). I improved the flamegraph PMDA late last year, but how FlameScope works
is really the direction Vector should go: being able to make an API call to
get the flame graph stacks as JSON, and then render them in d3, along with
buttons for transforms.

------
gnufx
What's the advantage of this sort of thing over the trace and profile analysis
tools that have been developed at least for HPC performance engineering over
many years? (E.g. the BSC tools
<[https://tools.bsc.es/>](https://tools.bsc.es/>) amongst several others.) In
particular, why are flame graphs preferred to the normal inclusive/exclusive
views of profiles which seem more generally useful to me?

------
petters
I know I am missing something, but passing -F 49 to perf will tell the kernel
to adjust the sampling interval dynamically so approximately 49 events/s are
recorded, right?

But then how does it make sense to plot the number of events per second in the
overview? Or what exactly is being plotted in the overview?

Edit: Perhaps it is simply that idle events are discarded and multiple CPUs
allow more than one event in the period?

------
lma21
Haven't followed up on this lately — is sampling java stacktraces with perf
going well? what happened to the frame pointer issue? do we still have to use
-XX:+PreserveFramePointer to get the full stack?

Awesome tool nevertheless. Always a pleasure to go through Brendan's utilities
regarding performance analysis!

~~~
brendangregg
Yes, perf sampling of Java is still going well, via -XX:+PreserveFramePointer.
Our one microservice that had high overhead with that option (up to 10%, which
is rare) has improved their code, lowered the overhead, and now enabled this
option by default.

But there's also a newer way that can do Java stack sampling from perf without
the frame pointer: [https://github.com/jvm-profiling-tools/async-
profiler](https://github.com/jvm-profiling-tools/async-profiler)

We're not running it yet. I want to try it out. Note that the stacks with
async-profiler are a bit broken -- Java methods become detached from the JVM
-- but I'm hoping that's fixable (it should be with a JVM change, at least).

------
hie2neiL
All the native linux thread stack visualizers I have used so far lack a way to
show single-threaded bottlenecks in multi-threaded applications. Does this
solve it?

In the java ecosystem this is solved with utilization timelines.

~~~
gnufx
I don't know which of the normal parallel performance tools support Java, but
at least TAU
<[https://www.cs.uoregon.edu/research/tau/docs.php>](https://www.cs.uoregon.edu/research/tau/docs.php>)
does; not that I've worked on Java, which should preferably be kept well away
from compute nodes.

~~~
hie2neiL
Oh, my question wasn't about java. Quite the opposite. I'm missing the perf
equivalent of some java tools I am accustomed to.

~~~
gnufx
I don't know what "perf equivalent" means. Parallel performance tools usually
use PAPI, which provides similar data, possibly a superset of perf data on
Linux. Score-P has an actual perf plugin. [http://www.vi-
hps.org/training/material/](http://www.vi-hps.org/training/material/) is one
source of material on such things, though biased towards distributed (MPI)
systems and OpenMP threading.

[Sorry I hadn't realized <> wasn't work as URL delimiters here, as I'm used to
elsewhere since the RFC <URL:...> convention got ignored.]

------
coding123
Pretty awesome - I recall a long time ago a few companies selling software
that would do similar (but different) things, so this can help us get closer.

------
sandGorgon
Given the existence of tools like this or
[https://github.com/uber/pyflame](https://github.com/uber/pyflame), I get
really amazed at the monopoly of NewRelic. Nobody seems to be even attempting
a start-up in this space, which is amazing given the amount of money that
everyone coughs up for NewRelic.

------
IamNotAtWork
What was the root cause of the problem that motivated this visualization tool?

~~~
brendangregg
It's described in the blog post under "Origin", and the sample profile that we
ship with FlameScope (examples/perf.stacks01) and which I used in the video is
the first minute of the original problem.

It was intermittent latency that was hard to explain using existing tools.

------
unictek
Can this be used with Node.js CPU Profiles?

~~~
brendangregg
Yes, I've already used it on Node.js profiles. :) Get any instructions for
doing CPU flame graphs with Linux perf (I've written several), and take the
output stage of "perf script" and you can browse it in FlameScope.

------
ProAm
I wish they wouldnt host this on Medium, the pop-ups and layout are maddening
and it almost seems unprofessional for a giant company like Netflix to use
Medium instead of its own blog platform on their website.

~~~
herrvogel-
Yes, Medium has terrible reading experience. I found this extension
([https://github.com/thebaer/MMRA](https://github.com/thebaer/MMRA)) a while
ago. You should try it.

~~~
oarsinsync
I keep reading comments like this, but I don’t ever see any particularly
glaring issues when reading from my iPad. Is it a desktop / chrome thing that
iOS / Safari is automatically mitigated against, or are my standards just so
low that I don’t notice the issues?

~~~
freehunter
Maybe they don’t do it on mobile but on desktop they have a pop-up demanding
you create an account to keep reading.

~~~
drusepth
Even when I open the link in an incognito window, I don't see this pop-up. Do
you need to be on the page for X time to see it?

~~~
kbenson
Second request in a certain period, I think. I opened this link twice. The
second time I got the pop-up, which starts out with something like "We've seen
you here before...". I just tried a third time and it did _not_ create a
popup.

As things go, I think they're actually handling that part fairly well.

