
Rreverrse Debugging - dbaupp
http://huonw.github.io/blog/2015/10/rreverse-debugging/
======
wyldfire
> I’ve never been a huge user of debuggers. Being able to diagnose my code
> line-by-line, statement-by-statement sounded theoretically good to me, but
> I’ve never really “clicked” with it in practice.

Years ago, I would've been floored by a comment like this. But lately I find
myself using them a little less often. These days, the codebases I interact
with leverage more and more small-scope unit tests. This usually means that an
unexpected test failure requires much less imagination to explain.

But debuggers remain a terribly simple way to identify the cause of a
segfault/bus error. Even with code I've never seen -- if armed with just a
stack trace I can identify a system misconfiguration or a workaround to avoid
the error.

~~~
dbaupp
I definitely use a debugger for getting a backtrace from a hard crash, but it
is (well, was) pretty much the only case where I'd reach for it as the first
choice.

~~~
yoklov
Have you ever used a good visual debugger? I find that command line debuggers
require you to keep too much state in your head at once, and even if you have
a great memory and this isn't a problem, they still require you to query the
new state after each step.

Sadly, most of the visual debuggers on linux don't meet my minimum standards
for "good" (crashy, slow, typically have trouble inspecting certain types
(mysteriously, since they're GDB frontends and GDB has no issue with it),
etc...), and successive update to MSVC makes it's debugger worse and worse
(although it's still the best I'm aware of at the moment)...

~~~
steveklabnik
I used visual debuggers in college, Eclipse and Java. But then I found Ruby,
and switched almost entirely from debugging to unit tests. Yes, they're not
entirely the same thing, but I had found that using a debugger meant I didn't
write tests. Once I found and fixed the problem, done.

Once I started writing tons of tests, I found I didn't need the debugger as
often. When your methods are less than ten lines, they're much easier to
reason about, and a method becomes a pretty decent 'unit' to check out, no
need to step through.

However, this also probably has to do with my relative skill levels and the
languages and tools involved too. Ruby debugging has gotten better, but only
in the last couple of years. And I didn't really know how to write tests
nearly as well when I was doing Java.

~~~
yoklov
I work in an area where unit testing is, generally speaking, more effort than
its worth (game programming -- the state space is generally too huge for unit
tests to be really practical). So I can't really comment on the effectiveness
of them, other than to say that yes, they are different.

Typically, instead of unit tests, I write state tracking and visualization
code. This goes a long way towards helping track down bugs and keeping me out
of the debugger, but honestly, a lot of that is just an indication of
debugging tools being crap in general.

------
pnathan
My master's work was in the context of debugging. I spent a great deal of time
reading descriptions of ancient debugging systems - most far in advance of the
capabilities of the present day. The height of the art was reached in the
early 90s - sophisticated logic structures over your program state were being
held over your program state to automatically trigger actions - reverse
debuggers were done back in the '70s as I recall; I believe the last major
advance was done by an Italian chap in '92\. Even SLIME and Common Lisp were
quite well passed up by these capabilities. Most of those advances took place
in the Lisp and Prolog world. I infer that this was due to the symbolic
execution capabilities which the Fortran/Algol families have disavowed.

So it's always nice seeing advances in the industrial art come out, but a
little wistful, knowing what once was.

~~~
EdiX
I would really like reading more about this.

------
pcwalton
To those saying "this has been done before in X language", "what's new in
this", etc., see the presentation:
[https://mozilla.github.io/rr/rr.html](https://mozilla.github.io/rr/rr.html)

The real interesting thing with rr is how it's done at the bare metal with
very low overhead, using deep hardware-level tricks involving the performance
counters on modern x86 CPUs. It's relatively easy to do this when you have a
VM, but doing it on native code running directly on the CPU is significantly
more interesting.

------
pron
For those interested, Java has a number of such "time-travelling" debuggers
(I've used one to great effect):

* TOD: [http://pleiad.cl/tod/index.html](http://pleiad.cl/tod/index.html)

* Chronon (commercial): [http://chrononsystems.com/](http://chrononsystems.com/)

* Jive: [http://www.cse.buffalo.edu/jive/](http://www.cse.buffalo.edu/jive/)

* Whyline (research): [http://www.cs.cmu.edu/~NatProg/whyline-java.html](http://www.cs.cmu.edu/~NatProg/whyline-java.html)

* Omniscient Debugger (discontinued?): [http://www.lambdacs.com/debugger/](http://www.lambdacs.com/debugger/)

~~~
karussell
Thanks for this list!

And even without: an IDE like NetBeans (and probably any major Java IDE) can
revert to a certain stack frame and even apply code changes on the fly - up to
a certain degree of freedom only, but okayish for normal programmer life,
where you need this only rarely if you have many smaller scope unit tests in
place.

------
MichaelMoser123
I just saw a presentation that describes the rr vm/recorder in more detail. I
am very impressed (though i haven't got the chance to use/test it).

[https://youtu.be/H4iNuufAe_8](https://youtu.be/H4iNuufAe_8)

pure science fiction - instead of recording the change produced by each
instruction they are recording what changed between system calls (result of
system call is recorded) and scheduling points.

They do record the result of each system call and somehow manage to count the
number of instructions per scheduled thread (they can only do that on new
intel processors - counting the number of branches not instructions as
instruction counter is not reliable. They are doing their own scheduling since
the VM is all running in a single thread. It is also very Linux specific - for
general case they use ptrace to record the system calls and their result, but
tracing of some operating system calls is optimized by injecting stuff into
the kernel (!)) - that's the reason why the recorded data is of minimal size.

I wonder how they are dealing with epoll - here the result is passed via
shared memory and not via the system call interface. Still i guess they are
lucky that there is no kqueue like interface on Linux - with kqueue it would
have been even harder to track when the event result comes in.

------
YorkianTones
Windows has had this capability for at least 8 years with "time travel
tracing" (or iDNA tracing), which produces a dump file that can be traversed
back and forward in WinDBG or Visual Studio. Very useful for figuring out what
caused unexpected state in complex integration scenarios with timing-dependent
bugs. [http://www.thewindowsclub.com/microsoft-time-travel-
tracing-...](http://www.thewindowsclub.com/microsoft-time-travel-tracing-
diagnostic)

~~~
moyix
iDNA traces are _huge_ though, and recording them adds a ton of overhead. rr
is way more lightweight.

------
padator
OCaml had a time-traveling debugger for 20 years now. Good job catching on.
(to be fair it got full stack back trace only since 2000 so far later than
Perl for that).

~~~
fulafel
GDB has had it for many years too. I think Mozilla made rr to handle large
apps better.

------
rjbwork
"Imagine"... I've been doing it in Visual Studio for years.

Step 1. Find problematic code

Step 2. Step back to before problematic statement

Step 3. Change problematic data value and/or code to hypothesized good one
(while debugger is active and paused)

Step 4. Continue execution and observe if output is as desired

Step 5. Write test case.

Easy peasy. I'm glad to see the Rust community doing something similar, as the
benefits of ease of development are not to be discounted, and Rust seems like
a pretty nifty language.

~~~
devit
That seems to describe memory editing plus "edit and continue", which is
distinct from record&replay and reverse execution.

Also, rr has nothing to do with Rust in particular, although this article
shows how to use it with Rust.

~~~
spott
rr works for C++/C code too?

~~~
nhaehnle
To expand on the monosyllabic sibling answer, rr is completely language
agnostic because it works at the level of machine code, (ab)using platform
features like performance counters to do its thing. It should basically work
for every x86 and x86-64 program you can debug with gdb, no matter which
language(s) it was compiled with.

------
alphabetam
IntelliJ's debugger can "drop frame", after which you can re-run a function.

------
ZenoArrow
This might be impractical, but is it possible to record gdb traces using a
different processor, as long as that processor has access to the RAM the
program is running in?

For example, could you record gdb traces with a GPU in a PC with hUMA?

[http://www.tomshardware.com/news/AMD-HSA-hUMA-
APU,22324.html](http://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html)

That could allow for gdb traces to be recorded constantly without a CPU
performance hit (traces could be truncated if/when the size grew too large).

------
barteklev
Yeah I love GDB. It's so mighty. The interesting that you can script your
debuging and it's just mindblowing. It helped me much in debuging net
communication. It was tough, but GDB let much! And It's for C/C++ too.

------
helmut_hed
A commercial product covering similar territory is UndoDB [http://undo-
software.com/undodb/](http://undo-software.com/undodb/)

------
fredcy
Elm has a "time traveling debugger", [http://debug.elm-
lang.org/](http://debug.elm-lang.org/)

I'm eager to give it a try in a real project.

------
arthursilva
Is there a good GUI wrapper for gdb or lldb?

~~~
josteink
Ive tried using gdb with Emacs (via GUD[1]) if that counts, but it never
worked out for me in the sense that I felt I got anything extra in return for
the investment.

Maybe there's some special things I forgot to do, but out of the box GUD
definitely felt (too) minimal for me.

Lately I've used LLDB when working on the .NET framework, and that seems like
quite a reasonable thing to work with too at this point. I'd love some emacs-
magic for this too :)

[1]
[http://www.emacswiki.org/emacs/GrandUnifiedDebugger](http://www.emacswiki.org/emacs/GrandUnifiedDebugger)

------
chris_wot
Oh man, this would have made my life troubleshooting a font layout issue in
LibreOffice one hell of a lot easier :-(

