The posture towards symbols on different platforms is quite interesting to see: I think all major OSes strip their binaries, but Microsoft provides them as a separate download and Linux of course has *-dbg packages and the upcoming debuginfod. In comparison, macOS is truly an infuriating platform because Apple strips their code and recommends others do so to (it's done automatically as part of deployment postprocessing) for size improvements but because Apple and third party developers are sold on the FUD that providing symbols will ruin their business nobody ever provides them separately. (Aside: if you ship software and run any sort of stripping/minification on it and want to claim it is for the size benefits exclusively, you are honor bound to provide the information you removed, whether that is DWARF or a source map. No exceptions. Personally, I consider this important enough to not do these things at all, but I get it if you fall on the other side of the "size vs. inspectability" spectrum–you just need to also provide the stuff you throw away if you don't want to be annoying.)
More on point, though, I can see both Windows and Linux getting the experience of finding debug symbols to be pretty good™–but for the same reasons I cannot see macOS getting this unless attitudes change. Apple of course runs this internally and "forgets" to share the scripts with us, and I have heard of other companies doing similar things, but I don't see any sort of unified effort in this area at all :(
That's not _wrong_, but I think it's misleading. Microsoft provides in such a way that they will automatically download on demand. That is fundamentally different from a manual download. Among other things it means that I can (and frequently do) debug crashes from other people's computers running different versions of Windows and I never have to think about it. It just works.
Oh, yes, totally; I just lumped it with Linux because the situation on macOS is so much worse that it's almost not comparable. I don't use Windows much, but from what I hear it seems like a dream to work with; Linux is ok once you figure out how to set up debug info (and I am hoping it can be like Windows in the future) but the situation on macOS is just hopeless because nobody is willing to share symbols. It's made even more sad because much of the infrastructure is already there; it's just locked up in private API and internal glue and with none of that crucial debug information to feed it.
There also used to be a special debug build of windows (and other MS software), which included debug symbols directly but also had various asserts etc. left in, specifically to enable debugging of issues that reached into OS components (for example when developing anything that integrated deeply into OS, whether in kernel or userspace).
Not sure if it's still provided but it was part of MSDN offering.
It's even more infuriating when problems you have involve system libraries. You can infer things from the available source code on opensource.apple.com, but it's not as useful as debug info. (also, the code for new versions of macos is only available weeks/months after they are released)
…and the source doesn't match with what they ship (often things get left out), and you still need to do reverse engineering to match it up with a stripped binary, and you can't easily load it into a debugger…yeah, it's not great at all.
I have compiled libraries from source and made them used with DYLD_LIBRARY_PATH. I haven't tried in a while but I think that doesn't work anymore without disabling SIP.
> Aside: if you ship software and run any sort of stripping/minification on it and want to claim it is for the size benefits exclusively, you are honor bound to provide the information you removed, whether that is DWARF or a source map. No exceptions.
Is that entirely fair?
Let's say you strip symbols to decrease size, but also in general don't consider inspectability to be particularly important (which isn't to say you're against it, merely neutral). You might decide that providing a source map simply isn't worth the effort, even if it's minimal.
Then you’re saying that you are OK with making the application opaque to all of your end users. You can claim you’re neutral on introspectability, but if you’re not doing the (minimal) work it takes to undo the information loss your net effect has been negative. Just to drive the point home, you can say you are “mildly for or neutral on efficiency” but then if you use Electron, it doesn’t matter what your position is, you’ve just made an bloated piece of software. And there isn’t even an easy fix for Electron…
Another related capability you can use when indexing PDBs for a build is the "source server". It tags the pdb with source control commands to retrieve the source revision at the time of building. Enable the feature in the debugger and you don't have to manually retrieve the right source code when you get a crash dump from a customer for an old version of your software.
It embeds executable source control commands in the PDB so you need to make sure you know where your PDBs come from.
Oh absolutely. Source-server/source-indexing is amazing. I use it dozens of times a week when debugging Chrome crash dumps. It's amazing when the correct version of the relevant source code just magically appears. I've blogged about this as well, of course:
As you say, knowing the provenance of your PDBs is important. That's part of why I pushed hard for https symbol servers a few years back. Don't ever use http: for a symbol server.
I really, really, really wish other platforms would embrace the same concept. We (Sentry) have added support for the same concept for other platforms but sadly there is not a lot of uptick in the general Linux community to start producing indexed debug symbols repositories.
It makes it easy to publish symbols for you .NET nuget packages. When debugging, Visual Studio will automatically download these symbols and the source code from GitHub, so you can easily step through third party code.
Having worked with Citrix a lot, it was always nice that both Microsoft and Citrix provide symbol servers. Despite not having any kernel debugging experience at all, it was always worthwhile setting up WinDbg with those two symbol paths to do some drive-by diagnostics of any blue-screen crash dumps. More than half the time this would instantly identify the "culprit".
More on point, though, I can see both Windows and Linux getting the experience of finding debug symbols to be pretty good™–but for the same reasons I cannot see macOS getting this unless attitudes change. Apple of course runs this internally and "forgets" to share the scripts with us, and I have heard of other companies doing similar things, but I don't see any sort of unified effort in this area at all :(