
Debugging Emacs, or How I Learned to Stop Worrying and Love DTrace (2018) - lelf
https://nullprogram.com/blog/2018/01/17/
======
masklinn
> Naturally I wanted to get my hands on this stuff to try it out for myself.
> Is it really as good as they say? Normally I stick to Linux, but it
> (generally) doesn’t have these Sun technologies.

Since then, bpftrace[0] has been released, supposedly it's actually better
than dtrace, in the sense that it's even more capable[1].

See [https://www.joyfulbikeshedding.com/blog/2019-01-31-full-
syst...](https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-
dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html) for a recent basic
ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF:
[http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-
traci...](http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html)

[0] [http://www.brendangregg.com/blog/2018-10-08/dtrace-for-
linux...](http://www.brendangregg.com/blog/2018-10-08/dtrace-for-
linux-2018.html)

[1] that's Brendan Gregg saying it:
[http://www.brendangregg.com/blog/2018-10-08/dtrace-for-
linux...](http://www.brendangregg.com/blog/2018-10-08/dtrace-for-
linux-2018.html) — having contributed to it and being one of the leading
dtrace experts

~~~
rurban
The extra capabilities Linux added made it insecure (arrays), so that eBPF
needs to be disabled in production SMP systems.

There's also no existing userland support. Everyone provides DTrace probes, no
one systemtap probes.

~~~
Birch-san
BPF is compatible with DTrace probes.

The kernel verifier does extensive bounds checks on array uses and jumps in
BPF programs.

~~~
rurban
Those checks don't save you from spectre-like cache exploitation. DTrace was
designed with maps-only for a reason.

Thanksfully there's now Oracle Linux with DTrace-proper.

When the usdt syntax can use static DTrace probes, why isn't it documented
then? and why doesn't it use the familiar DTrace syntax then? Why is nobody
using it?

~~~
dan-robertson
> DTrace was designed with maps-only for a reason.

You write this like you are implying that the DTrace designers somehow foresaw
that spectre would be a thing and therefore array accesses would be a risk.
This is obviously false. I’m sure there are reasonable reasons to not have
arrays but predicting spectre 10 years before it was found is not one of them.

~~~
rurban
Ask Brian, he did it on purpose.

------
agumonkey
Once again, a lovely article by Chris Wellons. Fun to see him having idols :)

ps: the hammer story linked at the end is worth the minute reading
[http://archive.is/OV1kQ](http://archive.is/OV1kQ)

~~~
nlefler
Seems likely the origin of the parable is Charles Proteus Steinmetz:

[https://www.smithsonianmag.com/history/charles-proteus-
stein...](https://www.smithsonianmag.com/history/charles-proteus-steinmetz-
the-wizard-of-schenectady-51912022/)

From the article:

""" Ford, whose electrical engineers couldn’t solve some problems they were
having with a gigantic generator, called Steinmetz in to the plant. Upon
arriving, Steinmetz rejected all assistance and asked only for a notebook,
pencil and cot. According to Scott, Steinmetz listened to the generator and
scribbled computations on the notepad for two straight days and nights. On the
second night, he asked for a ladder, climbed up the generator and made a chalk
mark on its side. Then he told Ford’s skeptical engineers to remove a plate at
the mark and replace sixteen windings from the field coil. They did, and the
generator performed to perfection.

Henry Ford was thrilled until he got an invoice from General Electric in the
amount of $10,000. Ford acknowledged Steinmetz’s success but balked at the
figure. He asked for an itemized bill.

Steinmetz, Scott wrote, responded personally to Ford’s request with the
following:

Making chalk mark on generator $1.

Knowing where to make mark $9,999. """

------
teacpde
For people like me who are not familiar with what line buffering is, I found a
good short explanation here
[https://www.gnu.org/software/libc/manual/html_node/Buffering...](https://www.gnu.org/software/libc/manual/html_node/Buffering-
Concepts.html)

------
drewg123
I'm a big fan of dtrace and I use it daily. However, the system call tracing
that the author did here could easily have been done without dtrace. Eg, via
strace on the curl process (or truss, ktrace, or whatever your OSe's system
call tracer is).

~~~
mjw1007
The author evidently didn't find that option attractive:

« To run curl under strace from Emacs, I’d have to modify Emacs’ behavior to
do so. With DTrace I can instrument every curl process without making a single
change to Emacs, and with negligible impact to Emacs. That’s a big deal. »

~~~
greglindahl
The usual trick to use strace in that kind of situation is to write a script
named "curl" that runs curl under strace, and stick the script in your path.

If the author is familiar with DTrace, though, sounds like that's a great
solution for them.

~~~
__david__
I don't see why "strace -f" wouldn't suffice. I use that all the time for
exactly these kinds of things.

~~~
ithkuil
Perhaps because you don't want to strace the whole emacs just to hit the curl
process that gets forked off emacs.

------
tedunangst
But the pty bug still hasn't been found?

~~~
josteink
That’s certainly how I read it.

------
m0zg
While we're on the topic of tracing, does anyone know an _easy to use_ (!)
tool that would roughly mirror the functionality of Apple Instruments for
user-space tracking? I know there's lttng-ust and TraceCompass, but using
either (and especially TraceCompas) is worse than pulling teeth. To display a
simple trace in a "piano roll" / Gantt chart view with TraceCompass one has to
read an encyclopedia sized tutorial and fumble their way through obscure XML
(!) files to create their own visualizations. Let's just say, ease of use did
not seem to be the goal at all.

------
ink_13
> Linux has a reputation for Not Invented Here (NIH) syndrome, and these
> licensing issues certainly contribute to that

I don't think the author understands with NIH means in practise, and this
completely misplaces the blame. It's not as though the kernel team was
completely unwilling to consider porting ZFS or DTrace, it's that for legal
reasons they were unable.

~~~
lelf
No (unfortunately) it's NIH and nothing more.

[https://youtu.be/TgmA48fILq8?t=4266](https://youtu.be/TgmA48fILq8?t=4266)

[https://marc.info/?l=linux-
kernel&m=154714516832389&w=2](https://marc.info/?l=linux-
kernel&m=154714516832389&w=2)

~~~
simcop2387
You've also missed what was said on the mailing list message. Sun specifically
made a new license to make it legally impossible to include DTrace and ZFS
(and other things) into a GPL licensed project.

That's a huge difference to what NIH means, which is that just because I
didn't make it, I'm not going to use it, I'll make my own version instead.

They couldn't ever use the original version of things because they're legally
bound to not do so. The kernel has had a long standing policy of not keeping
APIs that either they or userspace aren't using to reduce their burden and the
case of keeping that API _just_ for ZFS since literally nothing else in the
kernel was using it wasn't a good enough argument to keep it. ZFS on Linux has
now worked around the removal of the API.

~~~
pmoriarty
_" ZFS on Linux has now worked around the removal of the API."_

I'm just curious: how did they do that?

~~~
Arnavion
[https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f1...](https://github.com/zfsonlinux/zfs/commit/0c593296e98e3ac95f14704e4cee6cedb2134990)
\+
[https://github.com/zfsonlinux/zfs/commit/26a856594f731db6244...](https://github.com/zfsonlinux/zfs/commit/26a856594f731db62446cf537659f9360261fe97)

~~~
yjftsjthsd-h
> Linux 5.0 compat: Disable vector instructions on 5.0+ kernels

Isn't that going to kill performance?

~~~
simcop2387
Probably not going to be very noticeable, most of the cost in calculating the
checksums is in IO, but it will mean that there's a lower ceiling on total
possible performance.

That also assumes though that the cost for handling the vector instructions
in-kernel isn't higher than doing it without them. I believe that was part of
the reason that the kernel started removing this from within drivers and other
parts to begin with, the cost of saving all the floating point registers and
other state is actually surprisingly expensive. So it may be a complete wash
in the end anyway. If the kernel is no longer having to worry about saving
those registers it can mean that the userspace doesn't pay that extra cost and
now your compute ability is better for it without seriously affecting any of
the rest of the system.

