

Linux Performance Analysis: New Tools and Old Secrets - akerl_
http://www.brendangregg.com/blog/2015-03-17/linux-performance-analysis-perf-tools.html

======
mtanski
I've been trying to tell people to use perf for years now. I think there was a
lot of disservice done by early versions that were buggy, crash and freeze
prone. Still better then I oprofile before it.

It only got better once they added trace points to perf events. Now I can
simply trace all futex syscall enters to look for lock/synchronization
contention.

One extra tool I recommend is dynamic logging which you can enable from kernel
build. Create any log mask (file/like me/module) set a level and you're off to
the races. Invaluable in certain scenarios (like debugging bugs in fail path
of fscache module).

~~~
mtanski
Thanks to Brendan it's much easier to sell people on perf. You just link them
to Brendan's slides.

------
bcook
I am severely lacking in performance analysis skills.

Does anyone else spend hours tweaking builds/configs only to then just assume
you have made an improvement without confirming it?

I wish I had learned of Knuth's "premature optimization is the root of all
evil" quote sooner.

~~~
yeukhon
Same here. I think the hard part is choose what to control, how to measure,
what to capture. After all performance testing is running experiment.
Generating pretty visualization is 1% of the hard work if you have the data
you need...

------
planckscnst
These posts and talks by Brendan Gregg are awesome; they always prompt several
hours of playing around with new tools and digging into kernel code. I'm
trying to get familiar enough with these tools and methods that I can start
using them at work and spreading the knowledge around.

------
memming
Oh how fun it is to performance tune my data analyses

~~~
memming
"why is no one using ftrace already?"

~~~
autotune
I think you mean sysdig. All glory to the awesomeness and simplicity that is
sysdig: [http://www.sysdig.org/](http://www.sysdig.org/)

~~~
brendangregg
sysdig is simple, and has some awesome features, but also lacks many
capabilities (eg, kernel dynamic tracing) that we currently need ftrace or
perf_events for.

Edit: I should add -- there are three examples in the original post that use
ftrace. You can't currently do any of these with sysdig.

------
PublicEnemy111
This is so cool. Netflix seems to have a damn fine eng team. Its too bad they
don't have roles for new grads :P

Is there an authoritative beginners guide to linux performance analysis? A
cursory glance at trace-cmd tutorials shows I have a lot to learn. Or maybe it
would be more beneficial to start playing with a tool like sysdig?

~~~
cthalupa
Brendan's performance in the cloud/enterprise book is super solid. Not only
does it give information on performance analysis, but it's a very solid primer
to the systems involved. It's not specifically Linux - lots of solaris/illumos
stuff in it too - but it's definitely a starting point.

Not a ton on ftrace in it, though. Maybe we'll see a new book more focused on
Linux specifically with some ftrace stuff?

~~~
brendangregg
Thanks, yes, my Systems Performance book focused on the how and why of
performance analysis: background, concepts, architecture, methodologies. The
implementation specifics (which are an application of the earlier content)
were covered last in each chapter, which were Linux and Solari. I had some
perf_events but not ftrace.

I've been thinking of how best to cover my new ftrace and perf_events content
in book form...

------
specto
I've really been enjoying the use of sysdig for performance and other
analysis. It makes it quite easy to find what you need quickly. I do prefer
built in tools though, so I'll have to look at this.

