
What is BPF and why is it taking over Linux Performance Analysis? - okket
http://blog.memsql.com/bpf-linux-performance/
======
SFJulie
long story short. Average is a poor measure of non linear phenomenon like the
measure of a performance for a task .

Histogram are better to see the repartitions of values, and can hint you on a
n-modal behaviour. (a «mode» when everything goes well, and a degraded one for
instance which can hint on a potential transition of behavior).

I guess the next step is to say this is even more important to know the
evolution over time and announce that the equivalent of spectrogram (3D mapped
on 2D plus colour) is a good idea to known towards which mode you are tending
the more. [EDIT: which can also more significantly graphed as spectrum
analysis, colour encoding the amplitude of measure for a given frequency, and
frequency being 1/Period thus making sure that slow processes may appear
between 0 and average]

A shortcut is to hire of of the many jobless guys with a master in science
that actually have experience in labs.

Beware that for a reason that is beyond my reach, people with background in
math, management, literature have problems with scientific concepts that has
been discovered after 1800 unless it is packaged like magic.

~~~
static_noise
Replace the average (mean) with the median and you will have better results in
almost all cases.

When the data is good the result ist virtually identical. When there are
outliers, the median just ignores them unless there lots of them. When the
data is that bad, that the median doesn't work anymore you need to up your
statistics quite a few notches.

~~~
gnufx
Replacing a single number with a different one hardly helps. The distribution
matters, specifically the outliers in performance measurements.

------
vbernat
SystemTap enables the same kind of analysis for a few years already. And it
works well with older kernels (starting from 3.2, but really from 3.13). Of
course, an in-kernel solution is better (and I believe that SystemTap will
soon use it).

It's unfortunate that SystemTap never had much visbility. It also comes with a
huge libraries of tapsets.

~~~
josefbacik
Systemtap was fantastic, but super finnicky. You had to have debuginfo for the
kernel installed plus the source. This was fine for RHEL but kind of a pain
elsewhere. Then you run into other problems like sometimes the distro compiler
wasn't used to build the kernel you are running and suddenly you couldn't load
any stap script. Also the only way to get output was through stdout, there was
no easy way to programmatically pull results from an stap script.

BPF/BCC solves most of this. You still have to have the sources so you can get
the right targets for kprobes, but you don't have the compiler mismatch
problem. You can now access the hash maps directly from user space so you can
do things like build system monitoring tools that run in production and record
really specific information easily. I used systemtap for years, but bcc/bpf is
a whole new world.

~~~
paxcoder
You used past tense in talking about a project last committed to less than 3
days ago. I have no horse in this race but it seems to me like you might.

~~~
josefbacik
Or I just mistyped and you are reading in too much to it?

~~~
ignoramous
If you mistyped it, pls correct it.

~~~
okket
You can only edit a post for ~1 hour, his is 3 hours old...

------
forgotpwtomain
Low quality blogs designed to be used as marketing copy for the company. This
would be a much better link/resource:
[http://www.brendangregg.com/blog/2016-03-05/linux-bpf-
superp...](http://www.brendangregg.com/blog/2016-03-05/linux-bpf-
superpowers.html)

~~~
binarycrusader
That so called low quality blog is one Brendan himself linked to recently:
[https://twitter.com/brendangregg/status/766314203749638145](https://twitter.com/brendangregg/status/766314203749638145)

------
qwertyuiop924
eBPF could use a better interface, though. I REALLY don't want to have to
write this stuff in C. Think we could get a scripting language?

~~~
okket
You only have to write the kernel stuff in C (which is a small part, since you
are just collecting data), for the front end you can use BCC (Python/Lua)

[https://github.com/iovisor/bcc](https://github.com/iovisor/bcc)

Personal favorite:
[https://github.com/iovisor/bcc/blob/master/tools/sslsniff_ex...](https://github.com/iovisor/bcc/blob/master/tools/sslsniff_example.txt)

~~~
qwertyuiop924
Interesting. I'll have to read the docs, but...

This looks like a job for... FUNCTIONAL PROGRAMMING!

