
DTrace and Python - myautsai
https://github.com/pyconsk/2016-slides/blob/master/Python_and_DTrace/DTrace%20-%20pyconSK.pdf
======
bcantrill
Great deck! In particular, for the incredible magic on slide 14, a debt of
thanks is owed to John Levon[1] and to whomever has maintained that work and
brought it forward.

On slide 28, the presentation asks "What is 'Speculative Tracing'?" It's
unclear if that's a rhetorical question, but just to answer it here:
speculative tracing is a DTrace facility that allows for data to be traced
speculatively, and only committed to the trace buffer if and when some other
(later) condition is met.[2] My original inspiration for this was a case that
we had back in the day at Sun on the Performance and Application Engineering
(PAE) team, when Yufei Zhu (now at Facebook) described a case she had in which
one out of every 10,000 mmap()'s was failing with EINVAL -- and it was really
tough to use DTrace when she was only interested in its output 0.01% of the
time. For Yufei's (motivating) example, speculative tracing offered a way of
capturing all of the necessary data on every mmap request, but only emitting
that data when the entire operation was found to have failed. Speculative
tracing is one of those you-don't-need-it-until-you-need-it features of DTrace
(to which I would certainly add anonymous tracing) -- but when you need it,
it's a lifesaver, and I have it used it as recently as last week to nail a
particularly nasty bug that very much needed it.[3]

[1]
[https://blogs.oracle.com/levon/entry/python_and_dtrace_in_bu...](https://blogs.oracle.com/levon/entry/python_and_dtrace_in_build)

[2] [http://dtrace.org/guide/chp-spec.html](http://dtrace.org/guide/chp-
spec.html)

[3]
[https://twitter.com/bcantrill/status/769225926726918144](https://twitter.com/bcantrill/status/769225926726918144)

------
wslh
Shameless plug: if you are looking for something similar to DTrace but for
Windows that can be also used in Python or any other COM capable programming
language... you can take a look at our Deviare Hooking Engine:
[https://github.com/nektra/Deviare2](https://github.com/nektra/Deviare2)

Additionally, we have other open source instrumentation engines like
[https://github.com/nektra/Deviare-InProc](https://github.com/nektra/Deviare-
InProc) (better and more secure than Detours, check [1]), RemoteBridge:
[https://github.com/nektra/RemoteBridge](https://github.com/nektra/RemoteBridge).
SpyStudio will also be open sourced very soon:
[http://www.nektra.com/products/spystudio-api-
monitor/](http://www.nektra.com/products/spystudio-api-monitor/)

[1] [https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-
Cap...](https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-Captain-Hook-
Pirating-AVs-To-Bypass-Exploit-Mitigations.pdf) and
[https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-
Cap...](https://www.blackhat.com/docs/us-16/materials/us-16-Yavo-Captain-Hook-
Pirating-AVs-To-Bypass-Exploit-Mitigations-wp.pdf)

~~~
naveen99
interesting.

by the way, i got a google security warning on chrome trying to get to one of
the nektra blog pages: Google Safe Browsing recently found harmful programs on
blog.nektra.com.

also, have you considered an open source anti-virus program at nektra ? One
based on stuff like what's available here: [http://www.nsrl.nist.gov/nsrl-
faqs.html](http://www.nsrl.nist.gov/nsrl-faqs.html)

~~~
mxmauro
Hi @naveen99.

We upload precompiled samples but time to time Google marks us as harmful when
they aren't. Feel free to download the source code from
[https://github.com/nektra/](https://github.com/nektra/). Visual Studio
projects are provided too.

~~~
naveen99
Thanks. Will do.

------
pixelmonkey
Does anyone have any inside knowledge of when, if ever, DTrace might become a
standard/official part of Linux? Seems like right now you need to compile a
kernel module which will taint your standard kernel in order to use it.

~~~
gbrown_
> Does anyone have any inside knowledge of when, if ever, DTrace might become
> a standard/official part of Linux?

Never. Due to dtrace being licensed under the CDDL it will never be mainlined
into the upstream kernel. I would be keen to point out this does _not_ mean
there is a conflict of licenses from a legal point of view. Before anyone
starts making claims like "you're not allowed to do this", see the use of the
ZFS kernel module in recent releases of Ubuntu. I shall say no more on the
matter as a refuse to be drawn into a licensing debate...

Getting back on topic, those seeking more on user space providers should check
out this video from Bryan Cantrill.

[https://www.youtube.com/watch?v=6chLw2aodYQ](https://www.youtube.com/watch?v=6chLw2aodYQ)

Full disclosure I never had to run Solaris in production so I can look back on
Sun with rose tinted glasses. That said I still long for many of the features
it pioneered. Seriously ZFS would eliminate half of my production problems and
dtrace would make the remaining half solvable.

~~~
e12e
> I would be keen to point out this does not mean there is a conflict of
> licenses from a legal point of view. Before anyone starts making claims like
> "you're not allowed to do this", see the use of the ZFS kernel module in
> recent releases of Ubuntu.

AFAIK it's only packaged as a dkms-package:

    
    
      Package: zfs-dkms
      Version: 0.6.5.6-0ubuntu8
      Priority: extra
      Section: universe/kernel
      Source: zfs-linux
      Origin: Ubuntu
      Maintainer: Ubuntu Developers...
      Original-Maintainer: Darik Horn ...
      Bugs: https://bugs.launchpad...
      Installed-Size: 8 325 kB
      Provides: lustre-backend-fs,
        lzfs, lzfs-dkms
      Pre-Depends: spl-dkms (>= 0.6.5.6)
      Depends: dkms (>= 2.2.0.2), zfs-doc (= 0.6.5.6-0ubuntu8)
      Conflicts: lzfs, lzfs-dkms
      Replaces: lzfs, lzfs-dkms
      (...)
      xenial/universe amd64 Packages
    

Note that this is in "universe" _not_ Ubuntu proper.

From DKMS(8): dkms is a framework which allows kernel modules to be
dynamically built for each kernel on your system in a simplified and organized
fashion.

> I shall say no more on the matter as a refuse to be drawn into a licensing
> debate...

I really don't want to start that debate, but I also think the "Ubuntu
distributes (binary) ZFS" isn't quite accurate.

Now, how this all shakes out in terms of whether or not ZFS is bundled with
Ubuntu ("on the ISO so to speak"), and if it then really makes a difference
that it's distributed as source-code that can be automatically compiled -- I
don't know. I wouldn't think shipping pythons source (to be later compiled to
byte code, pyc) is enough to dodge GPL restrictions on bundling other
code/binaries as part of a distribution.

But, yeah, whichever it is, CDDL code isn't likely to ever be upstreamed to
the kernel proper.

~~~
lathiat
That's actually only half the story and mostly not relevant... the dkms
package is still there, but ZFS is now shipped with the kernel package:

lathiat@ubuntu:~$ dpkg -S zfs.ko linux-image-4.4.0-35-generic:
/lib/modules/4.4.0-35-generic/kernel/zfs/zfs/zfs.ko linux-
image-4.4.0-36-generic: /lib/modules/4.4.0-36-generic/kernel/zfs/zfs/zfs.ko

Announcement is here: [https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-
for-con...](https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-
containers-in-ubuntu-16-04/)

There is a lot of debate about the licensing legality of it, and there are
opposing positions from various parties. Here's the official post summarising
it: [http://blog.dustinkirkland.com/2016/02/zfs-licensing-and-
lin...](http://blog.dustinkirkland.com/2016/02/zfs-licensing-and-linux.html)

~~~
e12e
Thank you for taking the time to reply with those two links. Very interesting
(and directly helpful as one of my use-cases is led+zfs. Not needing the dkms
module makes things simpler).

------
dn0
Video:
[https://www.youtube.com/watch?v=HwqvHUGyGTE](https://www.youtube.com/watch?v=HwqvHUGyGTE)

