Hacker News new | past | comments | ask | show | jobs | submit login
Debugging Emacs, or How I Learned to Stop Worrying and Love DTrace (2018) (nullprogram.com)
258 points by lelf 81 days ago | hide | past | web | favorite | 32 comments

> Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies.

Since then, bpftrace[0] has been released, supposedly it's actually better than dtrace, in the sense that it's even more capable[1].

See https://www.joyfulbikeshedding.com/blog/2019-01-31-full-syst... for a recent basic ebpf/bpftrace tutorial. Brendan Gregg also has a primer on eBPF: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-traci...

[0] http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux...

[1] that's Brendan Gregg saying it: http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... — having contributed to it and being one of the leading dtrace experts

> * in the sense that it's even more capable*

Examples? I don't see how it's more capable. Catching up with dtrace, maybe.

bpftrace's potential is higher than DTrace's, but at the moment it's definitely not reached parity. For example: we cannot yet read strings bigger than about 240 bytes (any bigger won't fit on the BPF stack).

The extra capabilities Linux added made it insecure (arrays), so that eBPF needs to be disabled in production SMP systems.

There's also no existing userland support. Everyone provides DTrace probes, no one systemtap probes.

You can trace dtrace probes with ebpf.. see https://github.com/iovisor/bpftrace/blob/master/docs/referen...

BPF is compatible with DTrace probes.

The kernel verifier does extensive bounds checks on array uses and jumps in BPF programs.

Those checks don't save you from spectre-like cache exploitation. DTrace was designed with maps-only for a reason.

Thanksfully there's now Oracle Linux with DTrace-proper.

When the usdt syntax can use static DTrace probes, why isn't it documented then? and why doesn't it use the familiar DTrace syntax then? Why is nobody using it?

> DTrace was designed with maps-only for a reason.

You write this like you are implying that the DTrace designers somehow foresaw that spectre would be a thing and therefore array accesses would be a risk. This is obviously false. I’m sure there are reasonable reasons to not have arrays but predicting spectre 10 years before it was found is not one of them.

Ask Brian, he did it on purpose.

Once again, a lovely article by Chris Wellons. Fun to see him having idols :)

ps: the hammer story linked at the end is worth the minute reading http://archive.is/OV1kQ

Seems likely the origin of the parable is Charles Proteus Steinmetz:


From the article:

""" Ford, whose electrical engineers couldn’t solve some problems they were having with a gigantic generator, called Steinmetz in to the plant. Upon arriving, Steinmetz rejected all assistance and asked only for a notebook, pencil and cot. According to Scott, Steinmetz listened to the generator and scribbled computations on the notepad for two straight days and nights. On the second night, he asked for a ladder, climbed up the generator and made a chalk mark on its side. Then he told Ford’s skeptical engineers to remove a plate at the mark and replace sixteen windings from the field coil. They did, and the generator performed to perfection.

Henry Ford was thrilled until he got an invoice from General Electric in the amount of $10,000. Ford acknowledged Steinmetz’s success but balked at the figure. He asked for an itemized bill.

Steinmetz, Scott wrote, responded personally to Ford’s request with the following:

Making chalk mark on generator $1.

Knowing where to make mark $9,999. """

For people like me who are not familiar with what line buffering is, I found a good short explanation here https://www.gnu.org/software/libc/manual/html_node/Buffering...

I'm a big fan of dtrace and I use it daily. However, the system call tracing that the author did here could easily have been done without dtrace. Eg, via strace on the curl process (or truss, ktrace, or whatever your OSe's system call tracer is).

The author evidently didn't find that option attractive:

« To run curl under strace from Emacs, I’d have to modify Emacs’ behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That’s a big deal. »

The usual trick to use strace in that kind of situation is to write a script named "curl" that runs curl under strace, and stick the script in your path.

If the author is familiar with DTrace, though, sounds like that's a great solution for them.

I don't see why "strace -f" wouldn't suffice. I use that all the time for exactly these kinds of things.

Perhaps because you don't want to strace the whole emacs just to hit the curl process that gets forked off emacs.

Linux now has a number of tools that can do this just fine. strace was the only tool LOONG ago. For the issue described in the article sysdig would be the tool I'd use, but that's just one option.

But the pty bug still hasn't been found?

That’s certainly how I read it.

While we're on the topic of tracing, does anyone know an _easy to use_ (!) tool that would roughly mirror the functionality of Apple Instruments for user-space tracking? I know there's lttng-ust and TraceCompass, but using either (and especially TraceCompas) is worse than pulling teeth. To display a simple trace in a "piano roll" / Gantt chart view with TraceCompass one has to read an encyclopedia sized tutorial and fumble their way through obscure XML (!) files to create their own visualizations. Let's just say, ease of use did not seem to be the goal at all.

> Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that

I don't think the author understands with NIH means in practise, and this completely misplaces the blame. It's not as though the kernel team was completely unwilling to consider porting ZFS or DTrace, it's that for legal reasons they were unable.

The author also points out that "epoll" is a broken "kqueue".

There are many other examples of this.

For example, because of the broken way Linux did file locking, NFS was terribly broken on Linux for a very long time.

Linux is popular, but that doesn't always make it correct.

Sadly the way it has grown, many take Linuxisms as how every UNIX does it.

You've also missed what was said on the mailing list message. Sun specifically made a new license to make it legally impossible to include DTrace and ZFS (and other things) into a GPL licensed project.

That's a huge difference to what NIH means, which is that just because I didn't make it, I'm not going to use it, I'll make my own version instead.

They couldn't ever use the original version of things because they're legally bound to not do so. The kernel has had a long standing policy of not keeping APIs that either they or userspace aren't using to reduce their burden and the case of keeping that API just for ZFS since literally nothing else in the kernel was using it wasn't a good enough argument to keep it. ZFS on Linux has now worked around the removal of the API.

"ZFS on Linux has now worked around the removal of the API."

I'm just curious: how did they do that?

> Linux 5.0 compat: Disable vector instructions on 5.0+ kernels

Isn't that going to kill performance?

Probably not going to be very noticeable, most of the cost in calculating the checksums is in IO, but it will mean that there's a lower ceiling on total possible performance.

That also assumes though that the cost for handling the vector instructions in-kernel isn't higher than doing it without them. I believe that was part of the reason that the kernel started removing this from within drivers and other parts to begin with, the cost of saving all the floating point registers and other state is actually surprisingly expensive. So it may be a complete wash in the end anyway. If the kernel is no longer having to worry about saving those registers it can mean that the userspace doesn't pay that extra cost and now your compute ability is better for it without seriously affecting any of the rest of the system.

Linux guys did not invent xfs or jfs, yet they happily merged it. No NIH here. It is pretty silly to talk about that. If zfs license was compatible, they would have done the same.

The SystemTap people did not even try to copy any part of DTrace. Now the eBPF stuff is... pretty much a copy of the best parts of DTrace. So, yeah, it was NIH.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact