
Hist Triggers in Linux 4.7 - helper
http://www.brendangregg.com/blog/2016-06-08/linux-hist-triggers.html
======
cyphar
Glad to see that GNU/Linux is getting a DTrace-equivalent. Sucks that it's all
a bunch of string and staples rather than a single ioctl interface like DTrace
-- but hey, that's the Linux way.

~~~
netneutralish
Reminds me of developing countries that leapfrog industrialized G8 countries
in certain technologies, like skipping landline deployments and going straight
for wireless networks.

~~~
cm3
It's a shared medium and can never reach the qualities of wired networking,
though it might get better enough that we don't notice it as much as we do
now. This reminds me of efforts like xDSL G.fast. People only talk about
approaching the speeds of common fiber configurations, but miss the point of
very low latency in FTTH and easy upgrade to, say, 10 or 40 Gbs, whereas that
would require a technological enhancement in DSL or CABLE. With fibre it would
only require switching out the "modem" and network gear in the local uplink
station (hut).

~~~
Tiksi
Eh, you're not gonna get 10 or 40Gbps over most FTTH installs. Most run GPON
with beam splitters which maxes out at 2.5Gbps/line. To get 10/40/100 you need
OS1/2 singlemode or OM3/4 multimode fiber which I've never seen used in
residential/small business FTTP setups.

~~~
cm3
GPON isn't used everywhere, but it's used a lot, yes. Still, getting 2.5Gbps,
exclusively, with low latency isn't possible with the other connections
(yet?).

~~~
Tiksi
No doubt fiber is definitely the way to go if it's an option. Other than
fiber, the closest you'll get is DOCSIS 3 theoretically runs at ~1gig
downstream. Anecdotally, I've had Fios for a couple years and get faster than
advertised speeds consistently and never had an outage, so I'm all about
fiber.

I think it's due time that having an actual fiber drop becomes as common place
as having a phone line or cable line. With how much people use the internet
these days, it makes sense to have it's own connection instead of piggybacking
on whatever physical connection people already have.

------
cm3
Sorry, this is going to be rather long for HN, but it's been bugging me for a
while now, and this is a good opportunity to talk about it.

    
    
        Can't enhanced BPF do this too?
    
        Yes, if you program it to. I feel like we've been waiting for
        advanced tracing in Linux for years, and now two busses have
        arrived at the same time. This overlap concern was raised and
        discussed on lkml, and it was eventually deemed that hist triggers
        was different enough to be included.
    

This is what's wrong with Linux kernel development. We have pluggable
subsystems like I/O scheduling or CPU scheduling where BFQ or BFS are not
mainlined and have to live as external patchsets. Then we have more than a few
competing tracing/probing frameworks all somehow magically in
torvalds/linux.git despite their overlap and unfinished (partly experimental)
state.

I used to think that Linux kernel development is about who comes up with good,
working code, but more and more I'm starting to get the impression it's that
you need to have the right connections for a competing implementation to land.

I won't argue that linux needs DTrace or a similarly coherent solution,
although I absolutely believe that. However, seeing the impossible to explain
difference in what experimental, half-implemented code gets mainlined and
which widely used patches have to live outside the kernel, I cannot find
another explanation than requiring the right street cred and connections for
competing implementations to land. This is especially inexplicable for BFQ
because it's a in an already pluggable system.

To be clear, I'm not referring to projects like GRsec whose maintainers are
unwilling to split it into smaller chunks, and I'm primarily talking about
simple things like BFQ, which has solved my global lockups during large (long)
USB mass storage flushing. With BFQ, it's at most the application that called
fsync() and the rest of the applications are still responsive.

With all that being said, the evolutionary style of kernel subsystem
development doesn't work for everything, especially not for a DTrace
replacement, which would do well by having a single well thought out design
and broad availability of probes. While I can see that maybe people are afraid
of Oracle patents, DTrace isn't complicated enough that a similar effort
couldn't be implemented while keeping the same benefits. A simple transpiler
for DTrace scripts can be more than sufficient if the syntax is also preferred
to be different.

In summary, I don't understand how we have so many tracing system
alternatives, experimental and unfinished as well, while a pluggable subsystem
has only plugins personally accepted by that subsystem's maintainer.

~~~
brendangregg
Yes, there's two frameworks in the kernel that now have some overlap: ftrace
(incl. hist triggers) and perf_events (which can incl. BPF). But it's not a
case where they magically got included. Initially they didn't really have
overlap: ftrace was doing tracing, and perf_events was doing PMC stats. But
they grew over time (especially perf) to the point where there was some
overlap. This was noticed and has been discussed, and argued about. Eg, see
the later half of
[https://lwn.net/Articles/442113/](https://lwn.net/Articles/442113/). At some
point Steven Rostedt, ftrace author, had said:

> Now that perf has entered the tracing field, I would be happy to bring the
> two together. But we disagree on how to do that. I will not drop ftrace
> totally just to work on perf. There's too many users of ftrace that want
> enhancements, and I will still support that. The reason being is that I
> honestly do not believe that perf can do what these users want anytime in
> the near future (if at all). I will not abandon a successful project just
> because you feel that it is a fork.

Hist triggers is an enhancement to ftrace, not its own thing. BPF is a kernel
facility originally designed for network I/O, that can also be used by ftrace
and perf.

But yes, why not just one unified project? Such a project could bring in the
experts from the other external tracing projects too (SystemTap, LTTng, ktap,
sysdig, ...). Well, for better or for worse, Linux isn't a company, and Linus
isn't its CEO. Scott McNealy, CEO of Sun while DTrace was developed, was fond
of the saying:

> all the wood behind one arrow

For an April fool's prank, Sun staff put a giant wooden arrow through his
office window. [http://www.slideshare.net/brendangregg/from-dtrace-to-
linux/...](http://www.slideshare.net/brendangregg/from-dtrace-to-linux/34)

If Linux was actually Linux Microsystems, the CEO could pick one tracing
project and all teams would either toe the company line or leave. But we don't
have that.

What we have, including external tracers, is a mess. There's too many tracers.
And this becomes a tax on contributors who want to help fix the problem. Which
one should you pick? Better lean the pros and cons of them all, which is no
easy task.

Fortunately we're getting enhanced BPF, which does everything, so the tracing
mess will finally have an end. Hist triggers was a surprise. It would have
been a slam dunk before enhanced BPF existed, but now?? I suppose I understand
it as an ftrace enhancement, and not its own thing.

~~~
bcantrill
That's not the way that Sun worked at all, actually -- and certainly not with
respect to DTrace. To the contrary, Sun often deliberately allowed competing
projects, believing that the internal competition would generate a better
result. (Or, perhaps more likely, Sun simply did not have the command-and-
control to prevent such internal competition.) So contrary to your assertion,
there was never an executive fiat around DTrace (indeed, there was never much
executive fiat at all at Sun), and DTrace had loads of internal competition --
but DTrace succeeded where the others didn't because it was actually useful
for solving the problems that others weren't.

~~~
brendangregg
There's roughly ten different tracers for Linux. Yes, Sun liked competition,
but it did not endure this situation like this -- what project had ten
competitors, all staffed and paid for by Sun? If Linux were a company, I
cannot believe the situation would have gotten this out of hand.

~~~
bcantrill
There were (at least!) ten different kernel debuggers at peak in the late
1990s -- and many different tracers too. (As the saying goes, you haven't
heard of them for a reason.) My point is that Sun didn't get out of the morass
of myriad half-assed solutions by executive fiat; it got out of it by a small,
focused, self-motivated team that was hell-bent on building something that
could work on real problems. The nascent success of DTrace had nothing to do
with executive management, corporate heft, or marketing; these things only
were brought to bear when DTrace had already succeeded where it mattered most:
solving real problems.

~~~
brendangregg
A nice story, but it's not working out that way for Linux. The Linux kernel
engineering talent is there, but split among the different tracers, and, have
different priorities to work on. And this has dragged on for years. A rational
company could see that it wasn't working (no nascent success), and then
combine the right forces to get it done. Linux isn't a company.

