
LTTng: An open-source tracing framework for Linux - thezorg
http://lttng.org/
======
dmix
Wow, open source sites are getting nice designs these days. Is there a
business behind this project or is it via donation?

The documentation is excellent.

~~~
cbab
Thanks, we worked quite hard on this new redesign!

edit: Yes, EfficiOS [1] is the open source business behind the LTTng project
:).

[1] - [https://www.efficios.com/](https://www.efficios.com/)

~~~
dmix
Some UX feedback after attempting to install it on Fedora:

1) Clicked Fedora on install docs

2) Installed lttng-tools and lttng-ust RPMs

3) Clicked Getting Started in docs. Ran "lttng list --kernel".

3a) Got an error saying I need to run the systemd service.

4) Started systemd service (didn't see this mentioned in the docs?)

5) Reran "lttng list --kernel".... error saying I'm missing a module

5a) Reread docs, realized I need to build LTTng-modules. Which led me to find
out I need to also build liburcu

6) Installed liburcu but got an error building lttng-modules
[https://gist.github.com/dmix/d2d4c3c1b823e82486bf](https://gist.github.com/dmix/d2d4c3c1b823e82486bf)
(I could submit this as a bug report?)

So I gave up to attempt the install another time.

~~~
cbab
Thanks for reporting your issues on Fedora.

4) We will definitely fix that omission.

5a) liburcu is a dependency of the lttng-ust and lttng-tools. The dependency
should be accounted for in the respective Fedora packages.

6) Yes please submit a bug for this at [1]. We will investigate that failure
to build on Fedora.

[1] - [http://bugs.lttng.org/projects/lttng-
modules](http://bugs.lttng.org/projects/lttng-modules)

------
jpfr
LTTng is closely related to liburcu (now on
[http://urcu.so/](http://urcu.so/)).

It provides lockless datastructures for screaming fast concurrency on
multicore machines. You will be pleasantly surprised by the concurrent hash-
map implementation that comes with it [1].

[1] here's the paper for background:
[https://www.usenix.org/conference/usenixatc11/resizable-
scal...](https://www.usenix.org/conference/usenixatc11/resizable-scalable-
concurrent-hash-tables-relativistic-programming)

~~~
wrl
In the event LGPL is too restrictive, there's also concurrencykit
([http://concurrencykit.org/](http://concurrencykit.org/)), which provides
similar mechanisms (both for concurrent memory reclamation and various
datastructures built on top).

------
jordigh
Here is a pretty cool presentation about it, in French:

[https://www.youtube.com/watch?v=9XokHJadkq4](https://www.youtube.com/watch?v=9XokHJadkq4)

The slides are in English, if that helps:

[http://lttng.org/files/papers/presentations/lttng20tracingfo...](http://lttng.org/files/papers/presentations/lttng20tracingforeveryone.pdf)

~~~
cbab
You can also find all the presentations/publications that we did through the
years here [1].

[1] - [http://lttng.org/publications/](http://lttng.org/publications/)

------
rasur
So how does this compare to somerthing like Sysdig, for example?

~~~
mvlad
"Excluding proprietary solutions, a few competing software tracers exist for
Linux. ftrace is the de facto function tracer of the Linux kernel. strace is
able to record all system calls made by a user process. SystemTap is a Linux
kernel and user space tracer which uses custom user scripts to produce plain
text traces. sysdig also uses scripts, written in Lua, to trace and analyze
the Linux kernel.

The main distinctive features of LTTng is that it produces correlated kernel
and user space traces, as well as doing so with the lowest overhead amongst
other solutions. It produces trace files in the CTF format, an optimized file
format for production and analyses of multi-gigabyte data. LTTng is the result
of close to 10 years of active development by a community of passionate
developers. It is currently available on all major desktop and embedded Linux
distributions."

Further down you'll discover that your application(s) are linked with a
library provided by the project.

Most of them rely on kprobes [1], or like ftrace on [2]. But there is also
oprofile [3], and almighty perf [4], which rely on CPU extensions to implement
tracing, (also implemented and by the HW manufacturer as well -- but in a OS-
agnostic way).

[1] [http://lxr.free-
electrons.com/source/Documentation/kprobes.t...](http://lxr.free-
electrons.com/source/Documentation/kprobes.txt)

[2]
[http://lxr.freeelectrons.com/source/Documentation/filesystem...](http://lxr.freeelectrons.com/source/Documentation/filesystems/debugfs.txt)

[3]
[http://oprofile.sourceforge.net/doc/internals/index.html](http://oprofile.sourceforge.net/doc/internals/index.html)

[4]
[https://perf.wiki.kernel.org/index.php/Main_Page](https://perf.wiki.kernel.org/index.php/Main_Page)

------
hitlin37
how different is this from oprofile?

~~~
abusque
Well, I've never used OProfile myself, so please take this answer with a grain
of salt, but from a quick perusal of their website, and as the project's name
seems to suggest, OProfile is a sampling profiler, which usually means it will
collect data from counters at given intervals of time, producing a number of
statistics over time. This generally implies a somewhat significant overhead,
and consequent performance drop.

LTTng, on the other hand, is a tracer, which means it collects events from the
kernel (using the built-in tracepoint facilities) as they happen. It's also
possible to trace userspace apps (with lttng-ust), or define your own
tracepoints. This has the benefit of being much more detailed, and also has a
much smaller overhead.

You might be interested in reading the "What is Tracing?" section of the lttng
docs [0], which does a far better job at explaining this than I do.

[0] - [http://lttng.org/docs/#doc-nuts-and-bolts](http://lttng.org/docs/#doc-
nuts-and-bolts)

~~~
hitlin37
thanks for explaining. it sure looks interesting. i'm going to read now.

