
Mozilla's low-overhead open source replay debugger - bzbarsky
http://robert.ocallahan.org/2014/03/introducing-rr.html
======
dman
Another gem from Mozilla - I am astounded at how much good stuff is coming out
of Mozilla in such a short timespan. Keep it up folks. Between this, rust,
servo, asm.js and emscripten - you have my attention.

~~~
icarus127
I don't know about the author but not all of us oppose Gay marriage.

~~~
malandrew
Please don't confuse the personal views of the CEO outside of work with the
views of the entire organization. There are a lot of genuinely good people at
Mozilla doing good work, and I would bet that the overwhelming majority
working there do not share the same views as the CEO. You're comment not only
does a disservice to yourself and the HN community, but everyone else at
Mozilla working to make the world a better place.

------
fulafel
Was submitted yesterday:
[https://news.ycombinator.com/item?id=7458554](https://news.ycombinator.com/item?id=7458554)

It seems to me that a large portion of cool stuff doesn't make it out from
/new any more (not just this case, many others lately)

~~~
fijal
I was tracking a few of my own submissions - I think it's random. _IF_ you
happen to arrive on the frontpage and stay there for a few minutes, a
reasonable amount of people will vote that up. However, if you stay on the
front page for too short or not at all, the random fluctuation might kick you
down. This is e.g. raspberry pi foundation linking to pypy blog vs submission
of the pypy blog itself.

------
sanxiyn
[http://rr-project.org/rr.html](http://rr-project.org/rr.html) is a
presentation titled "How rr works" from Mozilla Research, with more details on
the implementation.

~~~
maninalift
...but that's great.

Wow, this is a wonderful thing.

I would go as far as to say it is the tool every developer wanted and most
thought was impossible (the low overhead part at least).

Only recording the nondeterministic parts and replaying the rest is so
obvious, once someone else has thought of it.

~~~
maninalift
except me of course cos I don't write bugs

------
nviennot
I've been very interested in that topic for the past years.

We've implemented an in-kernel linux multicore record/replay engine. See for
more: [http://velvetpulse.com/2012/11/27/scribe-the-
deterministic-t...](http://velvetpulse.com/2012/11/27/scribe-the-
deterministic-transparent-record-replay-engine/)

We also did a mutable replay engine, to replay on modified program (e.g.
replay a bug that happened in a production environment, but this time with the
logging verbosity turned into debug mode). Check out Dora:
[http://viennot.biz/asplos2013_dora.pdf](http://viennot.biz/asplos2013_dora.pdf)

------
gopalv
This is just a bit too awesome, yet simple.

Think of it as the "git" for debugging. Take a snapshot, keep going etc ...
and then backtrack back to the point where it all started going wrong.

Just very hard to automate easily, yet this uses gdb and all the other
macro/command goodness (well, for power users) that gdb comes with.

------
btown
An amazing project. Would it be possible to get to a certain point in a trace,
then switch the process back into "live mode" so that further inputs could be
made into the program? This could open up all sorts of interesting things,
like recording interactions with web applications and then allowing a
developer to open up high-level JS/DOM debugging tools to see exactly what
went wrong without needing to worry about replicating the exact input sequence
that led up to the bug.

It's unfortunate that the no-shared-memory limitation is so hard-baked in, so
it would seem that things like Chrome rendering processes wouldn't play nicely
with rr (I'm assuming they communicate with shared memory, correctly if I'm
wrong!).

~~~
sanxiyn
As I understand, you can't "go live" with the current architecture. In replay,
_no system call is executed_. Write is no-op. Read is done from the trace.
This is how PTRACE_SYSEMU works. To "go live", you would need to actually
execute system calls.

[http://sysemu.sourceforge.net/](http://sysemu.sourceforge.net/)

Re: Chromium IPC. "Channels are implemented using named pipes on Windows, and
socket pairs (or in some special cases unix domain sockets) on POSIX."

[http://src.chromium.org/svn/trunk/src/ipc/ipc_channel.h](http://src.chromium.org/svn/trunk/src/ipc/ipc_channel.h)

~~~
btown
Hmm, was afraid of that re: SYSEMU. Still, I'm sure people will find ways to
do higher-level debugging in rr with the current architecture!

------
moyix
This looks very cool, and will undoubtedly help with most run of the mill
crashes. Unfortunately the machine model it uses is single-core, so you're not
going to necessarily catch the really subtle concurrency bugs. Good multicore
record and replay is _hard_ , and has been the subject of tons of papers in
the past 10 years. It would be cool to see some of that research make it out
into practice.

------
wmat
This looks really interesting. I hope it doesn't get buried in the Facebook
buys Oculus news tsunami.

I can't wait to try rr out.

------
grundprinzip
"rr currently only supports x86 32-bit processes."

While the idea of a replay debugger is really nice, I don't understand where
this limitation comes from. I thought, that since years we practically live in
a 64 bit world.

~~~
chetanahuja
Yes. This. This sounds like a great project and I would love to integrate this
into our debugging workflow but this one limitation basically kills that
prospect.

~~~
iSnow
Well, it is version 1.0. If it is that important, maybe you could port it and
issues some pull requests.

------
e98cuenc
The latest gdb also contains a reversible mode. Anybody can comment on how
does it compare with rr?

I have only tried myself the reversible debugger in ocaml, and it was an eye
opening experience.

~~~
sanxiyn
The critical word is "low-overhead". gdb's implementation is orders of
magnitude slower.

On the other hand, rr can't do reverse debugging.

~~~
moyix
The lack of reverse debugging isn't an inherent limitation. One could index
the replay and take micro-checkpoints along the way; then when you want to
step backward, you rewind to the last checkpoint and play forward until you
hit the previous instruction.

------
haberman
Very cool! This looks a lot like something Michael Chastain was working on in
the late 90s:
[http://lwn.net/1999/0121/a/mec.html](http://lwn.net/1999/0121/a/mec.html)

It also is similar to GDB's record-and-reply functionality, except that I
believe GDB is limited to a single process. I suspect that GDB's approach is
much higher overhead also, because I think it actually records state changes
for every instruction: [http://sourceware.org/gdb/onlinedocs/gdb/Process-
Record-and-...](http://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-
Replay.html)

Unlike the GDB functionality, this doesn't seem to support reverse-stepping.
Hopefully this can be added -- it makes a huge difference when you can
reverse-step backwards from the failure to find out where things went wrong:
[https://sourceware.org/gdb/onlinedocs/gdb/Reverse-
Execution....](https://sourceware.org/gdb/onlinedocs/gdb/Reverse-
Execution.html#Reverse-Execution)

~~~
Tobu
In theory, the replayer could be made to rerecord a full memory trace. Running
the replayer in gdb would be enough for a single process.

~~~
haberman
> In theory, the replayer could be made to rerecord a full memory trace.

Yes exactly, or if it's cheaper, create periodic compressed snapshots and
replay from the previous one to recreate intermediate states. In my experience
GDB's recording is very expensive and very rapidly runs into resource limits.
Hopefully a snapshot-based approach would be more scalable.

------
Dorian-Marie
Link to the project: [http://rr-project.org/](http://rr-project.org/)

And the github: [https://github.com/mozilla/rr](https://github.com/mozilla/rr)

(On a side note, I think the gradient background is worse than a plain white
background, when removing it, it become a lot more readable)

------
userbinator
This has some similarities with a project I worked on that provided very
advanced analysis and debugging capabilities including instruction-level
"bidirectional execution". I can't say much more about that, other than the
fact that it was for a _whole-machine_ environment, and was a combination of
special hardware support, but it's good to see some of the techniques becoming
more widely available to the public.

------
jderick
This reminds me of a similar idea I'd like to see. A smart tracer that keeps
track of function call args and results and possibly local variables in a
cache. Then when your program crashes, it can show you not just the backtrace
but also recently called functions that may have influenced the crash. With
such a tool you may be able to debug some crashes without a debugger at all.

------
qwerta
Replay tool is also for Java. It is called Chronon, is Eclipse plugin and was
recently integrated into Idea Ultimate.

------
polskibus
Is this something similar to Intellitrace in Visual Studio?

~~~
sanxiyn
No. IntelliTrace records, but does not replay.

------
ArbitraryLimits
No one's going to rag on the post's author for labeling himself "Christian"
right at the top? He might have donated to Prop. 8 after all.

Edit: That's sarcasm, people.

~~~
gjm11
Sarcasm fully understood. So, to address the point you're making underneath
the snark:

No one (that I've seen, anyway) is ragging on Brendan Eich _for being a
Christian_. I don't even know whether he is, although it's a fair guess given
his decision to support Proposition 8.

Many people are saying that _Eich 's support of Proposition 8_ is a bad thing,
makes him a bad choice for CEO of Mozilla, discourages them from using
Mozilla's products, etc.

Attacking someone for being a Christian: religious bigotry based on
disapproving of someone simply on the basis of their identity. Generally a bad
thing.

Attacking someone for supporting Proposition 8: disapproval based on what the
person has _actually done_ and its likely consequences. No reason why it
shouldn't be a reasonable thing.

This doesn't seem to me to be a very subtle distinction.

(Aside: I do find it curious that of the four people who first come to mind
when I think of Mozilla -- Mitchell Baker, Gervaise Markham, Robert
O'Callahan, Brendan Eich -- two are rather vocally Christian (gerv's blog is
called "Hacking for Christ"!) and a third has become infamous for an action
that's probably the result of a similar religious position. Just coincidence?)

~~~
ArbitraryLimits
I'm confused, are you saying there's a link between people's beliefs and
actions or not?

~~~
gjm11
Of course there's a link. But criticizing an action someone has taken that
happens to have been religiously motivated is not the same thing as objecting
on principle to their religion.

An example more extreme than Brendan Eich's: Suppose someone is a devout
Christian but also rather deranged, and believes that the Bible is full of
coded messages telling him to kill (say) school biology teachers. So he goes
and kills a bunch of school biology teachers. If I say "That was a terrible
thing to do and he shouldn't have done it", that doesn't mean that I am going
to hate on everyone else who is a devout Christian -- because, as it happens,
plenty of people manage to be devout Christians without murdering biology
teachers.

Perhaps (so far as I know this is pure speculation) Brendan Eich is a
Christian and the particular variety of Christianity he adheres to tells him
to try to stop same-sex couples' relationships being legally recognized. If
so, then I think the particular variety of Christianity he adheres to is
leading him to do harmful things and I wish he would stop. But that doesn't
require me to object on principle to everyone who is a Christian, not least
because plenty of people manage to be Christians without trying to stop same-
sex couples' relationships being legally recognized.

Now, let us suppose there is a Christian denomination that has opposition to
same-sex marriage as a central part of its teaching, and that does not permit
anyone to belong to it unless they work to oppose same-sex marriage, including
making financial contributions to campaigns like the one for Proposition 8.
And suppose it turns out that some particular person is a member in good
standing of that denomination. Then I'm quite happy following the following
line of thought: This person belongs to that denomination; belonging to it
means being a fervent opponent of same-sex marriage and contributing to
campaigns against same-sex marriage; therefore this person has almost
certainly done things I consider harmful and antisocial; so much the worse for
this person. You may, if you please, consider this anti-religious bigotry. If
so, I invite you to consider the following two questions. (1) Is it possible
for a religion (or some particular version of a religion) to be morally
harmful to its practitioners? (2) If you learned that someone was a member in
good standing of an Islamic sect that had glorification of terrorism as a
central doctrine, would you think worse of them as a result?

