
Give me 15 minutes and I'll change your view of Linux tracing [video] - pmoriarty
https://www.youtube.com/watch?v=GsMs3n8CB6g
======
haberman
Argh. This talk reinforced my existing view of Linux tracing: it's really
fragmented.

Five years ago I tried to make some sense of this by researching all of the
existing technologies. In the kernel I found:

    
    
       - ftrace (https://lwn.net/Articles/290277/)
       - tracepoints (https://www.kernel.org/doc/Documentation/trace/tracepoints.txt)
       - kprobes (https://www.kernel.org/doc/Documentation/kprobes.txt)
       - events (https://www.kernel.org/doc/Documentation/trace/events.txt)
    

Now apparently we can add:

    
    
       - BPF, a packet filter that grew into a tracing framework
         (https://lwn.net/Articles/599755/)
    

In user-space we have:

    
    
       - perf
       - systemtap
       - lttng
       - other, random, fragmented things
    

This talk seems to add a bunch of other fragmented user-space tools.

I don't mean to put down anybody's work, but this stuff will never be user-
friendly as long as it remains so fragmented, IMHO.

~~~
xelxebar
I'm of a similar mind. However, somewhat recently I came across this article
which helped provide a framework to think about all these things. Turns it
that it's not just a flat space of competing tools:

[https://jvns.ca/blog/2017/07/05/linux-tracing-
systems/](https://jvns.ca/blog/2017/07/05/linux-tracing-systems/)

~~~
haberman
Excellent post, thanks!

------
erikb
okay, after 15 minutes of this.

Previous view: Linux tracing is so complicated. Without a personally important
usecase I probably won't invest the time to learn it.

Current view: Linux tracing is so complicated. Without a personally important
usecase I probably won't invest the time to learn it.

Soo... What I can say is I really hate these keyboard sounds. Otherwise I'm
not sure I learned much.

~~~
piyush_soni
And I'm jealous of his keyboard.

------
johnrivera
I can't help but giggle like a schoolchild because of those keyboard sounds.

~~~
pmoriarty
It was cute for about 2 seconds, then got incredibly annoying.

~~~
vacri
I get the feeling that it's more satisfying when it's on your own keyboard,
but it's definitely annoying on another person's.

~~~
ycombimike
I'll bet he can't even hear it.

~~~
justinjlynn
It's like playing a video game, I'd imagine. The sound effects are the bane of
everyone's existence but your own. Seriously though, you don't play your _game
boy_ in public without headphones; let alone present while playing one! I'm
amazed -- so, so tempted to turn of the preso even if I was fascinated by the
content.

------
lotyrin
BPF really seems nice. Ramifications to me though are: if I was willing to pay
a few percent overhead on all my production instances, what I would be able to
monitor 24/7 and get a return on the investment, and I haven't found much
writing in that area.

Seems like there could be a lot of opportunity, hopefully I'll get a chance to
dive in and find out myself.

~~~
tomsthumb
> I haven't found much writing in that area

This book would _probably_ get you moving in the right direction:
[http://www.brendangregg.com/sysperfbook.html](http://www.brendangregg.com/sysperfbook.html)

It should be something like: look at your bottlenecks and utilization, look at
your costs, look at (cost effective) ways to reduce or remove those
bottlenecks or that utilization. Pick the cheapest place to have a bottleneck.
Using SSD at an extra 30$ a month lets you use half the CPU and RAM, saving
60$ a month? Go for it.

------
pbhjpbhj
It was on his blog before, [http://www.brendangregg.com/blog/2016-12-27/linux-
tracing-in...](http://www.brendangregg.com/blog/2016-12-27/linux-tracing-
in-15-minutes.html) and has been posted here a couple of times. Interesting
this is the most traction it's got AFAICS.

------
Philipp__
Or just learn DTrace and hope it will be eventually ported _somehow_ to
Linux... /s

~~~
cryptonector
[https://github.com/dtrace4linux/linux](https://github.com/dtrace4linux/linux)

I... haven't tried it in ages. I've no idea if it works. It used to crash my
system easily, but maybe now it's fine.

------
viraptor
I feel like despite all this progress, sysdig is still the most accessible
solution at the moment. It even includes a slowish, but super simple way of
tracing user space (you can even write traces from bash scripts). I wish there
was a built-in Linux equivalent.

------
the8472
perf with source annotation is pretty nice if you're profiling for individual
hotspots. But I have not found any solution that lets me spot amdahl
bottlenecks which get drowned out in raw cycles spent by the parallel parts.
In java this is trivial with thread utilization timlines that incorporate
sampling.

Maybe this could be solved by weighting samples by the inverse of number of
running threads at the time

------
chicago_wade
Will BPF replace ftrace? From what I understood he was able to do everything
ftrace could by using BPF and BPF was more efficient.

~~~
cyphar
BPF was used to do the aggregation and calculation in-kernel. You still need
ftrace to actually run the BPF program in that context. You can read the cover
page for the patch that added this in 2015[1].

[1]: [https://lwn.net/Articles/630965/](https://lwn.net/Articles/630965/)

~~~
brendangregg
Right; plus theres some capabilities where ftrace is (and maybe always will
be) better. Eg, function counting: ftrace can count all kernel functions
instantly (try my perf-tools funccount tool), whereas the BPF method involves
setting a kprobe on everything, which takes much longer (setup and tear down).
And function graph tracing from ftrace will likely be better than anything we
can do in BPF (as it uses tracing all functions as well).

------
wyldfire
perf is awesome. ftrace is awesomer still for finding great stuff but I'm
often on a system with a kernel with no or limited support enabled for it.

