
Thoughts on Microsoft's Time-Travel Debugger - JoshTriplett
http://robert.ocallahan.org/2017/10/thoughts-on-microsofts-time-travel.html
======
themenace
This has parallels with the secret groundbreaking military technology that "60
Minutes"[1] was talking about in 2008. When you have unlimited capacity to
store data and the processing power to tag objects (or people!) in real-time
when you play it back, all kinds of new possibilities arise for software or
surveillance (good and evil).

My assumption is that the revolutionary advance was military aerial camera
systems (such as Angel Fire) that continuously record movement over an _entire
city_ in high resolution. When something happens, you can play it backwards to
see who drove or walked where, who met whom, find all the connections between
people, and trace events to the beginning.

[1]
[http://www.cbsnews.com/stories/2008/09/04/60minutes/main4415...](http://www.cbsnews.com/stories/2008/09/04/60minutes/main4415771.shtml)

~~~
euroclydon
Thanks for sharing that. Interesting what constitutes a military breakthrough
these days. A real nation-state with a military-industrial complex would just
shoot that thing down.

~~~
KirinDave
It's might become harder than you think.

Especially modern stuff. I've been talking with a lot of hobbyist drone owners
lately and several folks are working on fascinating projects using a fleet of
semi-autonomous lightweight fixed wing and quad drones along with modern
autonomous aircraft techniques.

These things are small (you could hold them one handed on a crowded bus), fast
(even the quads break 140mph), can be quiet (especially the fixed wings) and
have cellphones.

One of the projects I saw was proving that an automated drone network could
create a full coverage, high resolution video coverage of a parking lot even
with drones having to leave the site to get fresh batteries.

They're working for a team that has a DARPA grant, so...

~~~
zaroth
I'm surprised we don't see a lot more of this, but I assume it 10 years it
will be _extremely_ pervasive.

Drones which can zoom into an area at 100 mph plus and establish a complete
surveillance perimeter. They could dock on telephone poles throughout a city.
I imagine "calling 911" could dispatch these monitors to your location in
under 60 seconds on average.

Drones were a lot more prominent in the recent hurricane response, I think
it's the tip of the iceberg.

------
todd8
The idea of traveling forward and back in time in a debugger has been around
for a long time, even before the era of modern programming the idea was
explored and implemented. I recall conversations with people working on this
in the 1970's, but I never saw one working in action.

I've never had much use for debuggers. Of course I've had bugs that could have
been found faster with debuggers, but I usually find myself using logs of the
information I want to be quicker to work with. Of course this depends upon the
program and the environment one is debugging in. I'd rather reason about the
program statically than try to follow it dynamically in a debugger.

On the other hand, one of the best kernel programmers I ever worked with
carried his own debugger around and would implement it in a new kernel he was
working on just so he could have it at hand. (That's you Dammon if you're out
there!)

[1969] Rand's Extendable Debugging and Monitoring System,
[https://www.rand.org/content/dam/rand/pubs/research_memorand...](https://www.rand.org/content/dam/rand/pubs/research_memoranda/2009/RM5772.pdf)

[1988] Feldman, S.I. and Brown, C.B. (1988). Igor: A system for program
debugging via reversible execution.
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.85).
2238&rep=rep1&type=pdf

~~~
roca
The idea has been around a long time. Making it practical has been a much more
recent development. Ideas are cheap.

If you're using logs, you're using a debugger, just a very limited one.

Reasoning about a program statically is great, but once it gets above a
certain size you can't keep it all in your head and you must collect data to
narrow down the problem. That's where a debugger comes in. Then there are the
cases where you're debugging someone else's code and you don't have it in your
head in the first place.

~~~
roca
Oh yeah, and then there are the times when you reason about the program in
your head _and your reasoning is wrong_. That happens a lot. Sometimes it's
because of false but reasonable assumptions like "the compiler compiled my
code correctly" or "the CPU executed my program correctly".

------
mdriley
For more on the x86 emulation and tracing tech that powers this, see:
"Framework for Instruction-level Tracing and Analysis of Program Executions"
(2006)
[https://www.usenix.org/legacy/events/vee06/full_papers/p154-...](https://www.usenix.org/legacy/events/vee06/full_papers/p154-bhansali.pdf)

------
versteegen
Awesome, I hadn't heard of TTD. It's great to see reverse-debugging get
serious attention

rr sounds fantastic... unfortunately, I can't use it because AMD CPUs aren't
supported (they simply don't have adequate performance counters; it's not rr's
fault). With the exception of Ryzen, but even there, the performance counters
apparently aren't entirely reliable (meaning sometimes the debugger will go
haywire)

~~~
timmisiak
FYI, Microsoft TTD is supported on AMD CPUs.

------
j_s
Tools like this have always been how Microsoft segments the highest-price
"Ultimate" version of Visual Studio (or whatever MS marketing has renamed
things this month).

As best I can tell there is no mention in this article of how this feature
will be priced, which winds up limiting how often it will be useful simply
because of availability. Let me know when this shows up in the free SKUs
("Express") or perhaps even the drug-pusher-like "Community" (first hit is
free, unless/until you're making money) edition.

It is perfectly reasonable​ on their part as a company selling dev tools to do
this; I just feel like I'm window-shopping, unable to actually afford the
really cool bits. In my dreams I am way off base and this functionality is
freely available for all now - I kind of have the feeling something similar
been around for a while in the Ultimate SKU.

(Also, let me know if this ever comes to managed code.)

~~~
jasode
_> Also, let me know if this ever comes to managed code._

For managed code, VS2010 (Ultimate) introduced _Intellitrace_ for C# .NET
code. It let you step "backwards" which is a form of "time travel".

Since 2010, people have wondered if Microsoft was going to introduce a similar
debugging capability for C++. It looks like they did -- but instead of reusing
the previous "Intellitrace" brand and calling it _" Intellitrace for C++"_,
they've labeled it _" Time Travel Debugging"_.

~~~
SteveJS
Intellitrace and TTD take different approaches and have very different trade
offs. Intellitrace allows imposing logging on a system posthoc. TTD records
everything the system does. They are not merely labeled differently they are
different in kind. The consequences of those trade offs make the pragmatic use
cases very different. Intellitrace was always intended to head toward logging
in production systems, which i believe it has achieved now. It’s unlikely you
would TTD a server in production without a very good reason. It is possible
for a Intellitrace log to not have enough information to debug a problem. A
TTD trace has fundamentally captured everything, its bigger, a bit more
unwieldy... but in a deeper sense it is complete.

------
mandliya
As a sql server dev at microsoft, this has been lifeline for debugging
customer issues pretty quickly. It is slow without indexing, however once it
is indexed, it works really well.

------
jtl999
Very cool.

I seem to recall geohot doing similar to TTD in the past?

~~~
rjeli
[https://youtu.be/eGl6kpSajag](https://youtu.be/eGl6kpSajag)

~~~
Cyph0n
Damn, is he actually that humble or is it just a persona?

Skip to around 4:15 for the actual content.

~~~
kayoone
He's a smart kid and obviously likes himself to a degree that gets a bit
annoying but his achievements are quite something. I still find the
presentation style pretty entertaining.

------
pgh
For Java there’s the Omniscient debugger ...
[http://omniscientdebugger.github.io/](http://omniscientdebugger.github.io/)

------
dominotw
I could never get elm's time travelling debugger to work in practice. seemed
to make sense due to concept of time in FRP

[https://www.youtube.com/watch?v=vS3yzUo7l8Y](https://www.youtube.com/watch?v=vS3yzUo7l8Y)

hn discussion:
[https://news.ycombinator.com/item?id=7593032](https://news.ycombinator.com/item?id=7593032)

------
maxpert
As the article talk mentions it as well; I wonder how good it scales for long
running binaries and how well it integrates with stuff like .net native. I
mean doing artificial demos on a dll is cool, but how well will it scale for
say tracing bug in say a key-value store than runs for days and then crashes.

~~~
ComputerGuru
As far as I know, .NET Native isn't really something you're supposed to be
running for development; just a target you compile to for maximum performance
that should produce identical results (think of it as an uber-release mode).

~~~
pjmlp
Microsoft advises to regularly test .NET Native builds, to ensure UWP apps
work properly when compiled at the store level.

~~~
ComputerGuru
That’s correct. I just imagine the kind of cases that would need a time
travelling debugger _and_ only occur with .NET Native are not too many.

------
bcherny
Could someone help explain how this is different from a regular debugger that
lets you step back in the call stack? Does Chrome dev tools debugger (that
lets you step back in the call stack, even for asynchronous calls) come close
to TTD?

~~~
natoliniak
when you hit a break point in Chrome, you are suspending a live thread with
current memory and stack state. This tool is more like a profiler, that
instead of studying a current snapshot of the state, it will show you how the
state changed over time. (also, this is for native apps)

~~~
bcherny
I see. So for example, with a regular debugger you can see what the value of a
function's argument was at a point in the current call stack; with TTD you can
see what that argument was in previous call stacks too, and jump to those
other call stacks?

~~~
SteveJS
TDD allows you to recreate a full memory snapshot at an arbitrary point during
the recorded execution. A regular debugger cannot tell you what was in a
register or memory location previous to what is currently there now. TTD does
allow that.

~~~
bcherny
So:

\- Without TTD, I can inspect values in the call stack where I set a
breakpoint

\- With TTD, I can inspect values in my whole program, and can jump
back/forward in time to see how those values changed

Is that right?

~~~
timmisiak
Yup, you got it. Not just jumping back and forward, but seeking to points in
time when memory changed. For instance... have some memory that got corrupted?
With normal debugging you're probably out of luck unless you get some hints as
to where it got corrupted. With TTD, you just set a memory access breakpoint
and "run in reverse" until the memory is modified. It pinpoints the exact
point in time where the memory got corrupted.

------
maxdo
Redux also has sort of time traveling :)

------
partycoder
What I personally do not enjoy as a developer about MS products is that
eventually they hit end of life and you are left hanging. With open source
stuff, if the project is used enough there will always be some maintainer.

~~~
legulere
It seems to me like it’s the other way around. Microsoft keeps comparability
forever, which also has tons of negative consequences. With Linux it seems
almost like you have to recompile everything anew for each system and each
updates, where tiny differences in distributions can break things.

~~~
partycoder
Depends on your choice of distribution. Some distribution will distribute
compiled binaries for each architecture. And there are flatpaks that are
distribution agnostic.

For your own applications you will most likely use a container anyways and
that gives you fairly reproducible builds.

