Hacker News new | past | comments | ask | show | jobs | submit login
Thoughts on Microsoft's Time-Travel Debugger (ocallahan.org)
233 points by JoshTriplett on Oct 8, 2017 | hide | past | web | favorite | 72 comments

This has parallels with the secret groundbreaking military technology that "60 Minutes"[1] was talking about in 2008. When you have unlimited capacity to store data and the processing power to tag objects (or people!) in real-time when you play it back, all kinds of new possibilities arise for software or surveillance (good and evil).

My assumption is that the revolutionary advance was military aerial camera systems (such as Angel Fire) that continuously record movement over an entire city in high resolution. When something happens, you can play it backwards to see who drove or walked where, who met whom, find all the connections between people, and trace events to the beginning.

[1] http://www.cbsnews.com/stories/2008/09/04/60minutes/main4415...

Thanks for sharing that. Interesting what constitutes a military breakthrough these days. A real nation-state with a military-industrial complex would just shoot that thing down.

It's might become harder than you think.

Especially modern stuff. I've been talking with a lot of hobbyist drone owners lately and several folks are working on fascinating projects using a fleet of semi-autonomous lightweight fixed wing and quad drones along with modern autonomous aircraft techniques.

These things are small (you could hold them one handed on a crowded bus), fast (even the quads break 140mph), can be quiet (especially the fixed wings) and have cellphones.

One of the projects I saw was proving that an automated drone network could create a full coverage, high resolution video coverage of a parking lot even with drones having to leave the site to get fresh batteries.

They're working for a team that has a DARPA grant, so...

I'm surprised we don't see a lot more of this, but I assume it 10 years it will be extremely pervasive.

Drones which can zoom into an area at 100 mph plus and establish a complete surveillance perimeter. They could dock on telephone poles throughout a city. I imagine "calling 911" could dispatch these monitors to your location in under 60 seconds on average.

Drones were a lot more prominent in the recent hurricane response, I think it's the tip of the iceberg.

Interesting, but I’m sure there are significant efforts in anti-drone technology.

Breakthroughs are situational. Many first tier militaries spend time thinking about urban pacification. Within those contexts, it really is a breakthrough.

Just like how stealth aircraft were a breakthrough, and also totally useless in this situation.

Agree, but comparing it to the tank, in WW1 or WW2, where industrialized nations were fighting each other directly is flawed.

It's being used domestically in the US, more than a little concerning to me.

The idea of traveling forward and back in time in a debugger has been around for a long time, even before the era of modern programming the idea was explored and implemented. I recall conversations with people working on this in the 1970's, but I never saw one working in action.

I've never had much use for debuggers. Of course I've had bugs that could have been found faster with debuggers, but I usually find myself using logs of the information I want to be quicker to work with. Of course this depends upon the program and the environment one is debugging in. I'd rather reason about the program statically than try to follow it dynamically in a debugger.

On the other hand, one of the best kernel programmers I ever worked with carried his own debugger around and would implement it in a new kernel he was working on just so he could have it at hand. (That's you Dammon if you're out there!)

[1969] Rand's Extendable Debugging and Monitoring System, https://www.rand.org/content/dam/rand/pubs/research_memorand...

[1988] Feldman, S.I. and Brown, C.B. (1988). Igor: A system for program debugging via reversible execution. http://citeseerx.ist.psu.edu/viewdoc/download?doi= 2238&rep=rep1&type=pdf

The idea has been around a long time. Making it practical has been a much more recent development. Ideas are cheap.

If you're using logs, you're using a debugger, just a very limited one.

Reasoning about a program statically is great, but once it gets above a certain size you can't keep it all in your head and you must collect data to narrow down the problem. That's where a debugger comes in. Then there are the cases where you're debugging someone else's code and you don't have it in your head in the first place.

Oh yeah, and then there are the times when you reason about the program in your head and your reasoning is wrong. That happens a lot. Sometimes it's because of false but reasonable assumptions like "the compiler compiled my code correctly" or "the CPU executed my program correctly".

For more on the x86 emulation and tracing tech that powers this, see: "Framework for Instruction-level Tracing and Analysis of Program Executions" (2006) https://www.usenix.org/legacy/events/vee06/full_papers/p154-...

Awesome, I hadn't heard of TTD. It's great to see reverse-debugging get serious attention

rr sounds fantastic... unfortunately, I can't use it because AMD CPUs aren't supported (they simply don't have adequate performance counters; it's not rr's fault). With the exception of Ryzen, but even there, the performance counters apparently aren't entirely reliable (meaning sometimes the debugger will go haywire)

FYI, Microsoft TTD is supported on AMD CPUs.

not sure if undodb supports these... they do x86 and arm though so maybe?

Tools like this have always been how Microsoft segments the highest-price "Ultimate" version of Visual Studio (or whatever MS marketing has renamed things this month).

As best I can tell there is no mention in this article of how this feature will be priced, which winds up limiting how often it will be useful simply because of availability. Let me know when this shows up in the free SKUs ("Express") or perhaps even the drug-pusher-like "Community" (first hit is free, unless/until you're making money) edition.

It is perfectly reasonable​ on their part as a company selling dev tools to do this; I just feel like I'm window-shopping, unable to actually afford the really cool bits. In my dreams I am way off base and this functionality is freely available for all now - I kind of have the feeling something similar been around for a while in the Ultimate SKU.

(Also, let me know if this ever comes to managed code.)

> even the drug-pusher-like "Community" (first hit is free, unless/until you're making money) edition.

I mean, yeah it's not free-for-all, but "first hit is free" is kind of unfair. A company is not allowed to use it if they have more than "(a) more than 250 PCs or users or (b) one million U.S. dollars... in annual revenues" [1], and if it doesn't meet that criteria, it's usage is limited to 5 concurrent instances. More accurate would be "first hits are free, and you won't start getting charged until you're big enough where you can afford it".

Just making the distinction because I still have friends who think they can't use Community at home if they want to have a side project that makes money.

[1]: https://www.visualstudio.com/license-terms/mlt553321/

In case there's any confusion whether or not I agree with you, here's the very next thing I wrote:

> It is perfectly reasonable​ on their part as a company selling dev tools to do this

The Community edition licensing terms are, at best, awkward. I personally would have the hardest time with the 250 user limit - can I publish a free mobile app without fear this restriction might kick in?

It is not as blatant as the BizSpark / Imagine (DreamSpark) programs, so I guess that's a plus. The end result looks close enough to be considered the same to me.

Yep, it's annoying to have those limitations, but I sort of understand why. They could make their message a lot clearer though.

> I personally would have the hardest time with the 250 user limit

From the document, the text is "more than 250 PCs or users", so my understanding is that a company is limited if it has >= 250 computer users, not 250 customers. E.g. you could have a million free app users and it would be fine.

That fact that I had to preface that with "my understanding is that" means that the document is fuzzy, which is annoying, but at least further documents[2] can clarify a bit:

> - Any individual developer can use Visual Studio Community, to create their own free or paid apps. > - In non-enterprise organizations up to 5 users can use Visual Studio Community. In enterprise organizations (meaning those with >250 PCs or > $1M in annual revenue) no use is permitted for employees as well as contractors beyond... [the exceptions as noted].

[2]: https://www.visualstudio.com/wp-content/uploads/2017/03/Visu...

So you (as an individual not as part of a company) could create a paid or mobile app and have as many users as you want and make as much as you can. If you're a company, then you need to be under 250 PCs, <= $1M, and <= 5 Visual Studio users and you're still good. Exceed those numbers and you have to pay up.

It's freely available in the Windows 10 store as "WinDbg Preview". I recall reading somewhere that they will be rolling it into the Windows SDK once the preview is complete.

WinDbg can work with managed code through extensions, see https://docs.microsoft.com/en-us/windows-hardware/drivers/de....

>Also, let me know if this ever comes to managed code.

For managed code, VS2010 (Ultimate) introduced Intellitrace for C# .NET code. It let you step "backwards" which is a form of "time travel".

Since 2010, people have wondered if Microsoft was going to introduce a similar debugging capability for C++. It looks like they did -- but instead of reusing the previous "Intellitrace" brand and calling it "Intellitrace for C++", they've labeled it "Time Travel Debugging".

Intellitrace and TTD take different approaches and have very different trade offs. Intellitrace allows imposing logging on a system posthoc. TTD records everything the system does. They are not merely labeled differently they are different in kind. The consequences of those trade offs make the pragmatic use cases very different. Intellitrace was always intended to head toward logging in production systems, which i believe it has achieved now. It’s unlikely you would TTD a server in production without a very good reason. It is possible for a Intellitrace log to not have enough information to debug a problem. A TTD trace has fundamentally captured everything, its bigger, a bit more unwieldy... but in a deeper sense it is complete.

Thanks for straightening out my confusion!

TTD is a feature of WinDbg (free), not VS.

As a sql server dev at microsoft, this has been lifeline for debugging customer issues pretty quickly. It is slow without indexing, however once it is indexed, it works really well.

Very cool.

I seem to recall geohot doing similar to TTD in the past?

Damn, is he actually that humble or is it just a persona?

Skip to around 4:15 for the actual content.

He's a smart kid and obviously likes himself to a degree that gets a bit annoying but his achievements are quite something. I still find the presentation style pretty entertaining.

That's actually him. The stories of his shitty behavior when he was in college and Didn't Have To Do The Work Because Don't You Know Who I Am (which didn't work out so well) still circulate in my friend groups.

qira is cool but it doesn't scale. It singlesteps the program, recording the effects of each instruction in a fairly naive way. gdb's built-in reverse execution is similar. It's roughly a factor of 1000x slowdown and requires massive storage.

Getting the overhead down to < 2x (for rr) or 10x (for TTD), with reasonable trace sizes, requires much higher tech ... and is absolutely necessary for most users.

For Java there’s the Omniscient debugger ... http://omniscientdebugger.github.io/

I could never get elm's time travelling debugger to work in practice. seemed to make sense due to concept of time in FRP


hn discussion: https://news.ycombinator.com/item?id=7593032

As the article talk mentions it as well; I wonder how good it scales for long running binaries and how well it integrates with stuff like .net native. I mean doing artificial demos on a dll is cool, but how well will it scale for say tracing bug in say a key-value store than runs for days and then crashes.

As far as I know, .NET Native isn't really something you're supposed to be running for development; just a target you compile to for maximum performance that should produce identical results (think of it as an uber-release mode).

I’ve found and reported multiple critical bugs in .NET Native compiler. http://stackoverflow.com/q/34187890/126995 https://github.com/Const-me/EsentSerialize/commit/ea2cc7be58... The third one (only reported via e-mail), .NET Native fails to marshal custom COM interfaces implemented in C# classes to C++ libraries, at least on ARM for Win10 IoT.

Maybe .NET Native should produce identical results, but in practice it doesn’t always do that.

Microsoft advises to regularly test .NET Native builds, to ensure UWP apps work properly when compiled at the store level.

That’s correct. I just imagine the kind of cases that would need a time travelling debugger _and_ only occur with .NET Native are not too many.

Could someone help explain how this is different from a regular debugger that lets you step back in the call stack? Does Chrome dev tools debugger (that lets you step back in the call stack, even for asynchronous calls) come close to TTD?

when you hit a break point in Chrome, you are suspending a live thread with current memory and stack state. This tool is more like a profiler, that instead of studying a current snapshot of the state, it will show you how the state changed over time. (also, this is for native apps)

I see. So for example, with a regular debugger you can see what the value of a function's argument was at a point in the current call stack; with TTD you can see what that argument was in previous call stacks too, and jump to those other call stacks?

TDD allows you to recreate a full memory snapshot at an arbitrary point during the recorded execution. A regular debugger cannot tell you what was in a register or memory location previous to what is currently there now. TTD does allow that.


- Without TTD, I can inspect values in the call stack where I set a breakpoint

- With TTD, I can inspect values in my whole program, and can jump back/forward in time to see how those values changed

Is that right?

Yup, you got it. Not just jumping back and forward, but seeking to points in time when memory changed. For instance... have some memory that got corrupted? With normal debugging you're probably out of luck unless you get some hints as to where it got corrupted. With TTD, you just set a memory access breakpoint and "run in reverse" until the memory is modified. It pinpoints the exact point in time where the memory got corrupted.

Redux also has sort of time traveling :)


Since it seems as though you won't post civilly and substantively, we've banned this account.


What I personally do not enjoy as a developer about MS products is that eventually they hit end of life and you are left hanging. With open source stuff, if the project is used enough there will always be some maintainer.

It seems to me like it’s the other way around. Microsoft keeps comparability forever, which also has tons of negative consequences. With Linux it seems almost like you have to recompile everything anew for each system and each updates, where tiny differences in distributions can break things.

A statically compiled application will run for just about forever on linux. glibc has extensive backwards compatibility too. If you pack your application with its libraries, there's very little to worry about.

Kernel modules need constant recompilation but that's a very separate issue from programs.

Backwards compatibility across releases of the same product, sure. But some products hit their end of life and that's it.

If a product is deprecated, it may not receive security updates or fixes and such. As a user that puts you in a tough situation.

Unfortunately as a user there's not a lot you can do to influence the governance of those products or try to do something to keep them around. If the vendor decides to sunset a product that's it. That's all I am trying to say.

Depends on your choice of distribution. Some distribution will distribute compiled binaries for each architecture. And there are flatpaks that are distribution agnostic.

For your own applications you will most likely use a container anyways and that gives you fairly reproducible builds.

Besides the fact that you're simply not right (COM, MFC, ATL, SWF, WPF, etc are all deprecated yet still supported and maintained), how is this on topic?

COM used to be treated as a legacy technology by Microsoft, but it is pretty central now.

As extra info, COM became central after Longhorn's failure.

Many of the .NET APIs became COM based ones on Vista, with plain C Win32 ones being less and less common until we got WinRT, as the sibling comment points out.

Interesting enough WinRT shares many ideas with the genesis of .NET, before they decided to go the CLR route.

True, WinRT is built on COM.

Some examples: XNA, Silverlight, WCF, etc.

XNA was a victim of the WinDev going WinRT. Microsoft kind of excused themselves by sponsoring MonoGame for XBox ONE indies.

WCF is still being supported, there was a .NET Conf 2017 session.


If WCF is deprecated, what's the current preferred way for developing service-based applications?

Shit WPF is deprecated, whats the Microsoft preferred way to writes GUI apps on the desktop?

The flavor-of-the-day is UAP/UWP, which actually has awesome APIs. The only problem is that it's Windows 10+ and can't make universal or standalone binaries.

UWP also is like Silverlight more so in that it is missing a lot of stuff from the original WPF, you have to basically relearn it. However, Win2D is pretty good, nice performance.

> and can't make universal or standalone binaries

Highly ironic for a framework that's called Universal Windows Platform :D

Two different meanings of the word. I meant a single binary that can run on multiple versions of Windows. They mean universal as in one codebase that will deploy to PC, phone, and Xbox.


I know of better ways to kill my computer fan.

You're looking for trouble... But after thinking I have to second that.

Are you a .net developer??? Everything of MS still works, nothing has been deprecated. The comment you gave is really plain wrong.

WPF has seemed to basically been WinFormed, it still works, they might fix the bugs, but it isn't going anywhere and will eventually feel outdated.

The main problem with Microsoft in the past is that they didn't make money except by selling more software licenses. Things have begun to change with the cloud stuff, but the mentality is still there in the dev tools and frameworks - instead of "if it ain't broke, don't fix it" it winds up being "every time we move the cheese we make more money".

Microsoft's incentives still don't quite align with their developer/customers, but Microsoft does an amazing job pushing toward whatever is tossed out for sale as the new & shiny each year. The entire company is an amazing marketing machine.


Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? [...] The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features.

a) not true for reasons already given here b) they are(have) open sourcing a lot of developer stuff; I didn't check but would not surprise me if this ends up on github too if it didn't already with a good OSS license

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact