
BCC: Dynamic Tracing Tools for Linux - ingve
https://iovisor.github.io/bcc/
======
easytiger
For "Enterprise" environments it could take another 7 years to get to this
kernel level. Rather depressing. Many of my clients are still on RH 5.

~~~
sciurus
I hope you're working on moving them off it, since RHEL 5 reaches end-of-life
March 31, 2017.

~~~
devonkim
I suspect a lot of them would pull the same sort of thing as the IRS did with
Microsoft and pay millions for continued support specifically for them. Few
organizations would stay so far behind the curve without having a business
justification worth a substantial amount... or their IT costs are so
outrageous for the most basic levels of operation it's not too bad to pay half
your IT budget to avoid having to upgrade and retrain your likely demoralized
IT workforce and employees to handle an already-archaic OS like Windows 7.

------
ryao
These will be awesome once the newer kernels and LLVM versions that they need
are widely available. I am really looking forward to it.

~~~
drzaeus77
Hopefully you don't have to wait!

We're really hoping that this set of tools has only the minimal requirements,
that being the kernel.

Since Clang/LLVM can be linked as a static or dynamic library, we only have a
build-time dependency on those, and opt for the static library. The build
scripts have steps to build clang+llvm from source if your distro doesn't
include it. The binaries that we ship hide this and should (if we've done
things right) only have a libc and few misc .so dependencies. A change under
test even adds a fully static lua version.

On the kernel side, we don't conditionally compile anything based on kernel
features, and instead try/fail at runtime. You can just upgrade your kernel
and the tools will start to take advantage of the new features immediately.

------
scarmig
Naive questions to people in the know: is it the case that ebpf/bcc are the
standard for Linux tracing? Are they the Future(TM)? Or is the long-term
vision a pluralistic world of Linux tracers?

~~~
brendangregg
BPF (formally eBPF) provides the programmable tracing capabilities that Linux
previously lacked. Things like custom latency histograms. It works with other
built in features: kprobes, uprobes, tracepoints, etc.

So yes, BPF is becoming the standard and the future. It made it into the
kernel (well, the enhancements needed for tracing did; BPF has been around for
a long time as just a packet filter). In other words, BPF won. bcc is likely
the future, but we'll see other front ends as well (eg, ply).

~~~
brendangregg
I should add that Linux has two tracing front ends built in: ftrace and
perf_events. And perf_events is building BPF support (see lkml). bcc is
another front end for BPF. If this is confusing, I should categorize them like
this:

back end technologies: ftrace, kprobes, uprobes, tracepoints, BPF

front ends: ftrace, perf_events, bcc

ftrace is hard to categorize: it is a collection of both back end and front
end capabilities (although the front end is /sys). Also, it's being renamed to
just "trace".

~~~
scarmig
Thanks Brendan! A more navel-gazey question I believe you're well positioned
to answer:

For a long time, there seemed to be a conventional wisdom that Linux tracing
was mediocre, because it doesn't have DTrace. Has Linux approached or passed
the "caught up" point?

~~~
brendangregg
It hasn't caught up yet, but it's getting pretty close. Although I'd think
about it in terms of three dimensions:

\- technical capability: we should pass it this year. We'll be technical
capable of solving pretty much the same issues and then some.

\- ease of use for script developers: bcc is currently ~10x more verbose to
program. ply is much easier, but that project is in its early stages. This
won't matter that much for Netflix, since much of the time staff will use this
from a GUI (eg, Vector). It also won't matter that much for people who use the
many provided tools, like opensnoop, tcpconnect, ext4slower, etc. It matters
most for whoever codes the GUI and those tools (eg, me).

\- marketing, awareness, brand: Sun marketing, sales, trainers, and
evangelists, did an awesome job - everyone knows about DTrace, even if they've
never used it. In comparison, bcc/BPF (and many other Linux kernel
technologies, like ftrace) have virtually zero marketing budget or
professional marketing staff promoting them. Awareness will take the longest
to catch up.

I also summarized the current state of bcc/BPF in
[http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-
ro...](http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-
ahead-2016.html)

~~~
brendangregg
... I should add, a tipping point for awareness may be when a commercial
performance monitoring company adds BPF support (and latency heat maps, etc),
and then promotes this feature. There's a lot of performance monitoring
companies out there, many are well funded, and have professional
marketing/sales departments.

------
rurban
I couldn't find a debian repo, and I wonder what to do with the bcc nameclash
for the source package. bcc is an existing tiny c compiler in debian.

