
Bpftrace for Linux 2018 - deirdres
http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html
======
pzakha
Hi Brendan, thanks for the update and the work on this! Coming from dtrace,
the main thing missing in bpftrace when doing kernel debugging is being able
to navigate structs.

You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if
it was leveraged in bpftrace, but it looks like it isn't yet. I see you've
opened an issue in one of the bpftrace forks related to this here:
[https://github.com/ajor/bpftrace/issues/30](https://github.com/ajor/bpftrace/issues/30).

Do you know if there's been any update on this? Support for BTF/CTF would be
amazing.

~~~
brendangregg
That's the old repo (we should add a note to it pointing people to
[https://github.com/iovisor/bpftrace](https://github.com/iovisor/bpftrace)
instead!)

Alastair added struct support for kprobes yesterday, based on the
functionality in bcc (which bpftrace uses). That was the final missing piece,
and why I'm posting about it now. See the last example here:

[https://github.com/iovisor/bpftrace/blob/master/docs/referen...](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#2-kprobekretprobe-
dynamic-tracing-kernel-level-arguments)

It's the same struct support we had in bcc, using linux kernel headers. Some
things are available, some are not. Just like in bcc, we sometimes need to
define some of a missing struct.

BTF will mean that all structs are available, plus, bpftrace should be able to
automatically figure out the arguments so you don't have to cast them. It'll
be great, but in the meantime, I am able to do tons of stuff.

~~~
pzakha
Fair enough. Having to include headers is one thing I found tricky with BCC,
as there's sometimes non-trivial dependencies between various header files,
especially for kernel modules that aren't included in the mainline kernel.
That said I'm very excited to see all the progress that's been done in
bpftrace, and looking forward to what's to come.

------
keeler
This is huge. The Linux community should be glad that Netflix hired Brendan
Gregg to work on Linux performance.

~~~
moetech
Is this because they are also using Linux or do they plan to use it on
FreeBSD?

~~~
mburns
Those aren’t mutually exclusive options.

~~~
gregdunn
For now they are, and that might hold true in the future as well.

There are some projects to bring BPF features needed (generally called eBPF)
to FreeBSD, such as [https://github.com/YutaroHayakawa/generic-
ebpf](https://github.com/YutaroHayakawa/generic-ebpf) \- but there's no
accepted mainline implementation as of yet.

------
WhatIsDukkha
Seeing this come together makes me super happy.

Now the next time the hot rage of an audio xrun overcomes me I will have a
path forward!

In general audio xruns actually is an area that could use a permanent
userspace program.

Musicians need low latency audio processing which Jack2 can provide but the
random xrun and the internet flimflam/mummery of how to sort it out is waaaaay
beyond most of us (I have less of an excuse but yah its a vibe killer when you
are trying to make music).

The goal would be "hey you had a/some xrun and its might be mostly this disk
or process or..."

Maybe this needs hard ai but maybe it just needs a compass like tool...

------
Birch-san
This is a great update. I'd wondered how ply differed from bpftrace. And
structs support is a huge step in achieving parity with DTrace.

DTrace has really helped me out* on macOS. To be without it is to lose a
superpower, so I'm really rooting for bpftrace to bring the same tracing
experience to Linux. I've been excitedly following bcc, too. It's just the
right tool for HTTP metrics. But my most basic experiments so far have failed,
and I don't know how to get support.

* DTrace is a real swiss army-knife. It seems intended for observing production systems, but I've found it indispensible for local development. The mysqld provider is great for tracing queries and perf. Opensnoop answers all sorts of questions. And DTrace saved a C++ project of mine: explaining a deadlock that would've otherwise spelled the end of the software. The most fun I've had is using the pid provider to hook text out of visual novels (in WiNE!).

~~~
gregdunn
>But my most basic experiments so far have failed, and I don't know how to get
support.

I'd recommend the #iovisor irc channel on oftc. I haven't stopped by in a bit,
but the times I did, I usually was able to get assistance from nice people!

------
gregdunn
Hey Brendan! Super exciting - I'm already updating some internal documentation
to point people towards this :)

When are we getting a Linux performance engineering book with all the
bpf/bcc/bpftrace stuff in it?

------
pmoriarty
Is there any chance this tool or maybe a wrapper for it could get a higher
level, less C-like syntax?

~~~
helper
The syntax seems to be inspired by awk more than c.

------
lelf
1\. Obviously Linux-only. How is it DTrace 2.0?

2\. That’s a staggering amount of NIH, from CTF to syntax. Why?

~~~
brendangregg
1\. There's work to bring eBPF to other operating systems.

2\. bpftrace is built from the ground-up to operate with eBPF, which does much
more than DTrace. I made a list in the post, and that's only happened in the
last two years. I understand that if I'm narrowing eBPF to just DTrace vs
bpftrace comparisons, you might question why we're doing all this, but
consider the bigger picture of eBPF.

------
rurban
The 2 major problems (ie the Elefant in the room) compared to DTrace are
missing:

1\. security: eBPF still supports arrays and and is therefore inherently
insecure to be used in the kernel. DTrace didn't support that feature on
purpose and not because they couldn't do it. see spectre/meltdown.

2\. availability: eBPF probes only work within the kernel. DTrace extends the
same interface to the libc level and the user space level. you can trace not
only kernel calls, also lib calls and with its high-level language support
(Java, python, perl, ruby,...) into userspace. Linux-only eBPF will never be
able to support that seemless integration. thanksfully we now have Oracle
Linux with proper DTrace support on Linux.

~~~
efefef3
Hmm, looks like you are missing some crucial details, eBPF has the same
mitigations to the CPU bugs in place as the rest of the kernel (e.g.
retpolines on tail calls, index masking for arrays, sanitation of speculative
store bypass, etc). Also, eBPF probes do not only work in the kernel but user
space as well via uprobes as one example.

~~~
rurban
Of course they ran after possible array problems post-spectre, but
nevertheless it still allows precise timings on anything cache related. it's
still highly insecure and needs to be turned off in production. and this is
where it makes sense to use.

2nd: do you see uprobes supported anywhere? I still can only use DTrace with
high level languages. and I wouldn't use this island solution anyway. DTrace
works everywhere, now even on windows. eBPF is Linux-only. And should have
stayed netfilter only BTW.

