Hacker News new | comments | show | ask | jobs | submit login
Leaving Mozilla (ocallahan.org)
258 points by curtis on Mar 2, 2016 | hide | past | web | favorite | 103 comments

I've never heard about rr http://rr-project.org/), his tool to record program execution and debug it afterwards. The alleged 1.2x slowdown in execution is negligible. The ability to reverse in time is almost a nirvana. I'm sad I didn't have it when I used to program C++. Hope he succeeds in his new endeavor. We'd all win big.

Does anyone knows of similar tools for Python and in-browser Javascript?

Mozilla is prototyping a feature called "WebReplay" to support reverse-debugging of JavaScript and DOM mutation:



It looks like Microsoft is implementing a similar time-travel feature in Chakra:


For Python my coworker (and friend) Tom has a great project called timetravelpdb: http://everythingsysadmin.com/2014/04/time-travel-pdb.html explains a bit on it, the repo is https://github.com/TomOnTime/timetravelpdb

Basically it's an extension of pdb that allows you to jump back and forth in time.

Re: Javascript - If you use the'redux' (http://redux.js.org/) framework you can get something like this (https://github.com/gaearon/redux-devtools-dock-monitor)

I understand you never claimed they are similar. But to be fair (and with due credit to redux for being a great framework), all redux does is save method call arguments in an array. rr is on a different scale altogether.

I was really impressed when finding out about Microsoft's Chakra JS engine Time travelling debugger[1], I think that's what you're looking for, haven't tried it myself though.

[1] https://youtu.be/1bfDB3YPHFI?t=304

If I'm understanding this correctly, then Java has had it for ages now via the "Chronon" debugger. It's the only reason why I pay for an IntelliJ Ultimate Edition license, since it comes bundled.

There are other reversible debuggers, but even a mature one like Chronon has more overhead than this. 1.2x is basically nothing.

Not if you have > 1 busy thread.

No exactly the same thing but Tonic has similar capabilities using CRIU (checkpoint and restore in user space) for Javascript time traveling: http://blog.tonicdev.com/2015/09/10/time-traveling-in-node.j...

It's an interesting different approach because it allows you to rewind the entire machine, so you can interact with interactions between different processes, etc (happy to answer questions, I am a dev on it)

If you're willing to accept things that compile to in-browser javascript, check out Elm's magical time-traveling debugger: http://debug.elm-lang.org/

I'm building something like this for in-browser Javascript, but in a larger context. We're not ready for public consumption yet, but: https://fabric.fm

Interesting! Tried to subscribe but got an error: {"code":-5000}

the python case is interesting ... after all, the interpreter is just another binary executable. perhaps there's a way to write a layer above rr to "step" in Python code rather than the underlying C code. same with JS (Node.js) or any other language run-time.

Seems rr is using fork for its implementation. The first time I've seen this approach taken is in Ocaml's debugger.[1]

[1] http://caml.inria.fr/pub/docs/manual-ocaml/debugger.html

Sounds very similar Green Hills Software's TimeMachine for embedded applications.

TimeMachine needs a specialized hardware probe connected via JTAG, if I'm not mistaken. The probe is powerful enough to record everything (registers, operations, etc) in its memory without affecting the embedded device. Then you can replay the info however you want, in as much detail as you like.

The feature here is more like software emulation, and it will really allow you to take the instruction pointer in reverse direction. Unlike TimeMachine which records in real time and let's you browse the recorded data as you wish, but after execution has occurred.

My thoughts were very similar to yours. In 1992, when I started out in embedded on a Philips 8051 and debugging it was via an ICE and had direct visibility and control of the 8051.

The ICE would be connected to a bonded out version of the 8051 and I was almost able to capture every single clock cycle of 8051's execution. When I hit a failure, I would just replay the trace and see what went wrong.

It looks like the folks on the big machines have caught up :-)

I had access to a Huntsville Microsystems ICe that used a bog-standard - non-bonded-out - 68008 in harness. I had scripting tools that would capture and replay all the registers to back the machine up.

That HMI was a brilliant machine.

It's really the last debug tool I ever used besides AVR, PIC and one ARM JTAG for doing BSP work.

Python and in-browser Javascript are also programs, in theory rr covers the entirety? But I don't know what the ergonomics are like (e.g., how much will you see program activity in your platform that you don't care about, and how well can you see the activity in the code you are debugging?)

Yes, rr can record python and JS execution just fine, but the debugging experience is hopeless, since right now rr just gives you gdb. With an astounding tolerance for pain you can make progress by calling JS engine functions from gdb, but nobody wants that.

This is fixable, however, and on my agenda.

There is something similar in bpython called rewind. Run a few commands in the repl and then try `ctrl+r`.

I wonder how it works. Assuming it must capture state at t=n, and somehow calculate the diff to get to t=n+1. Could theoretically pop diffs to get to any point in the past.

Its not free, but there is http://undo-software.com/undodb/

Gdb has supported reverse debugging since 2009.

I've used it, and I've been using rr. gdb's reverse debugging was useful to me a few times, in extremely limited situations.

I often do my initial runs under rr, just in case I might want to debug, because it's useful most of the time and it's not enough overhead to worry about.

gdb's reverse debugging works after you narrow the range of interest down to a very small region, because otherwise its buffer will repeatedly overflow and it'll take ages.

I worked with Robert at Mozilla for many years. He might be the most talented software engineer I've ever met, and he's a strong candidate for nicest person as well. An absolute pleasure to work with. Mozilla will miss him dearly but some other project (apparently rr) just got really lucky.

rr is roc's baby ;-).

From reddit to answer the question some might pose ("who?"):

> One of Mozilla's most senior engineers who has worked on Mozilla projects as a contributor and employee for well over a decade particularly working in the graphics code and helping make native audio and video in the browser a reality.


Great! Now how about the next question:


I suppose it's nice to know that he is a big deal but as someone who doesn't follow mozilla religiously a little more background is always appreciated.

Why is he leaving? That's covered in the OP: he wants to devote his full-time attention to working on rr, a reversible debugger for native code that imposes impressively low overhead, and possibly spin it into a startup.

I think roc would give me a sly smile for pointing out that this doesn't really answer the "why". It's what he's going to do after he leaves. Doesn't quite answer why he's leaving.

Because... he wants to do those things? I'm really unsure what you're trying to say.

Well, it'd be foolish to assume that someone doesn't have multiple reasons for making a dramatic change in their life.

It'd also be foolish to assume that most of those reasons are any of our business.

Robert is an amazing guy, and we're all lucky to have benefited from his work over the years. Now, he's set off on a new adventure, for reasons of his own... And has hinted that he'll continue sharing some of the fruits of his labor.

That's more than enough ;)

He just wants to build a startup out of some cool tech and get rich. There's no conspiracy or dirty laundry here.

My, cynical much? Seems to me that 10+ years at a non-profit, and choosing to live in a place outside the traditional startup locations (New Zealand) with fewer networking opportunities, might indicate fairly little interest in that exact trajectory, for those exact reasons.

But what do I know, I'm just another ten-year Mozillian myself. And if or when I leave, it'll also surely be to get filthy rich at a startup, right?

And for those that have no idea what rr is: http://rr-project.org/

So no, he is not leaving for "rest and relaxation"

To be fair, people also think "rest and relaxation" is amazing, and we probably need to capture some of the value of this technology via a for-profit company as well.

I would pay to be able to time travel and replay my rest and relaxation!

Wow, it's the end of an era.

I wish him the best of luck. rr is really awesome and deserves even more attention.


Regarding RR he says:

> we probably need to capture some of the value of this technology via a for-profit company

I hope this doesn't mean making it proprietary. I mean obviously a person gets to decide how to best compensate themselves for the fruits of their labor. But I've seen too many compelling technologies fail to achieve the impact they could have because they were proprietary.

I mean, DTrace languishes despite being open-source, because it wasn't open-source enough

There are multiple ways to fail at spreading technology. One way is to choose a business model that makes it difficult for potential users to actually use the technology. Another way is to choose a business model that doesn't generate enough revenue to bring the technology to its full potential.

Collabora does for-profit work on LibreOffice, but integrates their work into LibreOffice's master. Mostly they even do this fairly openly and don't batch up commits then push them but make their changes small(ish) then push them.

You can definitely be a for-profit company AND work in the open.

There's a decent market for selling consulting consulting services on top of open source technology. Being the expert on applying an open source technology to solve specific problems comes with a decent rate card.

You're still selling your labour rather than a product, though. It's possible to be well off doing that, but basically impossible to become ludicrously rich.

He didn't say his goal is to become ludicrously rich. He said he has ideas that Mozilla can't and shouldn't provide resources to realize, so some for profit company would work better for it.

True. There is a potential multiplier when you're selling a product or money, rather than time. It's a matter of priorities and risk appetite.

http://undo-software.com/undodb/ is a rival which i have marketed at me on linkedin frequently

DTrace doesn't languish. It just doesn't have a Linux port, because of people's contrived rewriting of the history of the GPL.

Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

This is a technology that received an innovation award from the Wall Street Journal in 2006. Whatever impact DTrace is having today, it should have been much larger.

I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day.

DTrace is used extensively in FreeBSD development these days.

It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/

My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.

We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.

I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.

I use dtrace often, but am too busy to write articles about it, sorry. A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.

I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel.

For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.

You really aught to look at Bryan Cantrill's twitter feed then. It happens frequently enough that he notices blog posts and tweets them.

Then again Cantrill has an axe to grind.

In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill

I'm no fan of the man, but it doesn't mean that dtrace is not an excellent tool.

We use it every day at Circonus. One of us is learning something new about how our stuff works every single day.

Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX.

It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.

I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same:

    /pattern/ { statements }
In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations).

So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.

This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.

> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

Then you haven't been looking:








Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace

Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)

It does, it's part of Oracle Linux. Oracle obviously therefore believe that a CDDL kernel module is fine.

It wouldn't be distributed as a kernel module (DTrace probes are far more invasive than that). But yes, I posted this before I realised that Oracle Linux has sublicensed it under the GPL in their port to Linux (for their customers), so it's fine for us to do the same as long as we "base it on their port".

Oracle also owns SUN, who originally licensed it that way...

Thanks Robert for signing up for StackOverflow just to answer my question http://stackoverflow.com/q/34488237

I'm not familiar with Robert or his work, but I quite liked this post. The brevity and honesty was refreshing. I couldn't help but feel excited at the end!

Reverse debugging has been around for several years, but hasn't seemed to be widely adopted. Which seems kind of unfortunate, as the concept of it looked really useful.

Does rr make reverse debugging much easier than it's previously been in gdb or something? :)

I've tried to use rr on LibreOffice but had no luck whatsoever. :(

Please file an rr issue. I've had it working in the past.

Just saw this - will do!

rr (http://rr-project.org/) is a piece of magic. I've heard lots of people talk about using it in wild and awesome ways.

The second part of a magic trick is to connect it to your audience. Making time-travel style debugging is... not easy, but possible. Making it relevant to people, across languages and other conceptual domains, is hard.

If your next work is tackling this, good luck and godspeed :)

Great work; Great song!

Wonder if that's the end of Mozilla's only southern hemisphere office in Auckland.

No. Mozilla's Auckland office has other engineers working on Gecko video playback.

I know of at least one other NZ Mozillian (Nick Cameron, on the Rust project) but I'm not sure if he works remotely or at an office.

There were about ten NZ Mozilla employees when I worked there a year ago. Not all at the Auckland office but there were plenty there to keep the office utilized at the time.

Pretty sure he lives in Wellington and so must work remotely.

If and only if there is a way to debug network traffic like I am stopping at this router and hop to the next router.

"we probably need to capture some of the value of this technology via a for-profit company"


Perhaps they don't want to end up having to beg [0, 1] for donations so that they can maintain a high quality application while also being able to afford trivial niceties like "food" and "housing."

[0] https://www.propublica.org/article/the-worlds-email-encrypti...

[1] http://arstechnica.com/information-technology/2014/04/tech-g...

Because there's a ton of work to do and getting it done in a reasonable timespan is going to require some talented people to work on it full-time.


Everything you said there is incorrect.


Religious flamewars, which you predictably set off with this, are their own circle of off-topic hell on Hacker News. Please don't do this here.

Christians of all denominations actively proselytise - it's a core part of being a Christian, spreading the 'good news'. This distinguishes the Christian ethos from most other mainstream religions which have a weak or non-existent missionary instinct.

So no, it's not unusual at all for a Christian to highlight the fact. From the point of view of a Christian in an increasingly secular world it probably feels more necessary than ever.

P.S. I am a Christian by birth but not vocation.

To be clear, only some strains of Christianity are active proselytizing. There are strains that do none at all.

Most strains of Christianity will have some form of proselytizing, though it might be muted. Plus there also tends to be ebbs and flows in the history and culture of such things. It's interesting to note that proselytizing arguably has a very significant (positive) effect on formation of liberal democracies [1].

[1]: http://isites.harvard.edu/fs/docs/icb.topic965299.files/Wood...

Christian by birth? You can't be a religion by birth. Your parents were Christian when you were born but that does not make you a Christian by birth.

It's the bio in his personal blog. He can put whatever he wants in it. And I don't think he includes it because it is "fashionable", but rather a reality that a true faith defines who you are and how you operate.

It's a personal blog, and that's a bio. A short list of interests, things that might turn up in the blog.

Is it different from calling oneself an abstract impressionist? Or a functional programmer? Christian teachings can absolutely be applied in software. I mean, look no further than the Temple operating system. Christianity is not a "skin" on that project, it's intrinsic to its design.

>I mean, look no further than the Temple operating system.

Ughh. One would hope you would pick a better example than workings of a confirmed schizophrenic with an anger management issue who "talks to God".

If I were to pick an example it would be something like Larry Wall's Perl treatises.

Since the beginnings of religion?

And I'm curious, where does it "define" him? he posted a Hillsong song.

No respect for your comment I'm afraid, as it shows your own personal biases very clearly.

Then you see it's Hillsong, and it all begins to make sense.

If you don't know what Hillsong is... well.. Wikipedia. :)


I'm not entirely sure what you think isn't offensive about your comment.

1. You are responding to the comment "Since when has it become fashionable to define oneself by religion?" (aside: the answer is since forever) in the affirmative

2. You find it "interesting" that "clearly rational and intelligent people have faith", by which you say that most people who have faith (and by association, Christians) are irrational and stupid.

3. You say that you believe that you expect those who have faith must be seriously uneducated.

4. You say that people who have faith use it as a crutch.

Hope this helps.

Being offended.

I am offended by you saying that people who have faith aren't irrational. That goes against my worldview and is therefore offensive to me.

Irrationality is cognition, thinking, talking or acting without inclusion of rationality. It is more specifically described as an action or opinion given through inadequate use of reason, emotional distress, or cognitive deficiency.

i wonder; do many people usually get offended merely by some abstract sense of "going against my worldview"?

Yes. That is exactly what being offended by verbal attacks to your religion is.

"verbal attacks to your religion" can mean many things, including abusive and disrespectful behavior, so it's unclear.


I have no problem with you sharing an observation. I do have an issue with you denigrating people who have faith.

Yep, I should stop sharing opinions on things that I simply fail to understand on any level.

I'm a bit touchy about this subject as I'm a Christian myself, so sorry if my response was a bit heated.

I can't reply directly to your apology, but there is no need to apologise.

Good Karma all around.

Yes, per the blog post he plans to spend more time working on rr than he thinks it's reasonable for Mozilla to have to support:

And now there's rr. People think it's amazing, and it is, but we're only scratching the surface of what could be done. I'm bursting with ideas, and realizing them will require a lot of resources, resources that Mozilla cannot and should not provide --- we probably need to capture some of the value of this technology via a for-profit company. We can change the way people debug software, and in its own way that may be as important as my Web platform work, and it's work I desperately want to do.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact