I used to work on a legacy piece of business software, written mostly in C. One recurring issue we noticed about GDB was that compiling a "debug" version of the binary to be able to GDB it often fixed whatever problem the program was having, such that we were no longer able to reproduce the issue. So it wasn't uncommon for me to have this conversation with my manager:
Me: Customer is having their program X hit a SEGFAULT when they run it.
Manager: Did you transfer over a GDB version to backtrace the core dump?
Me: Yep.
Manager: What did it say?
Me: Well, when I run the same series of steps through the GDB version, it doesn't segfault, so I have nothing to backtrace.
Manager: Huh...
Me: What would you like me to do?
Manager: Move the production binary aside, drop in the GDB version. Leave it there.
Heisenbugs[1] can be incredibly frustrating. Computers are supposed to be deterministic. Even when bugs are incredibly complex[2], it's possible to systematically investigate iff the behavior of the system is deterministic.
I once had to add 7 NOP instructions at the beginning of the bootstrap/"BIOS" code I wrote for a Z80 clone. I couldn't understand why test programs seemed to crash[3] about a 15% of the time. Later, I discovered the same behavior in code that previously worked. I spent over two weeks trying to investigate, which only produced more confusion as the behavior would sometimes go away or get much worse randomly with each change I made.
I finally found the bug using a (hardware) logic analyzer to watch[4] what the CPU was doing on the memory buss, Something wasn't finished resetting inside the CPU. Any instructions that ran too early would trash the internal state of the CPU, causing later instructions to have problems like asserting multiple chip select pins. Multiple ROM/RAM chips would try to drive the buss, and everything dies. The instruction this happened on depended on which instructions were run while the CPU was still resetting. The NOPs simply delay startup to let the CPU stabilize.
Bugs disappearing in debug builds is common to all C and C++ codebases. It's the nature of undefined behavior and it's interaction with modern optimizers.
I'm in the camp that says all builds should at least be built with symbols, so you can still debug release builds. Maybe stripped out of the distributed version, keeping a private copy for coredumps from the field, if you're doing closed source dev.
Here's what I've done: First, you should have a tagged version in the source code control system for everything you ever ship. That should include the tools used by the build process. So you should be able to reproduce the shipped version to the byte.
Second, gdb can give you a stack trace with addresses if there are no symbols. But if you compiled with gdb, you can give pass the flags "-Wl,-M" to the link step, and it will spew a link map out stdout, which you can pipe somewhere. With that file, you can figure out what the addresses in the stack map correspond to.
If you really want to go hard core, once you figure out what function the problem is in, you can compile that file with "-Wa,-ahlms=<filename>.asm.out" to get an assembly listing for that file. From a bit of hex arithmetic, you can find the assembly instruction that corresponds to the crash. The hard part is correlating that back to the source code, since I don't know of a way in gdb to get that assembly output with the C/C++ source as comments.
> The hard part is correlating that back to the source code, since I don't know of a way in gdb to get that assembly output with the C/C++ source as comments.
If you run debug builds, the compiler fetches an uninitialized variable out of memory that the OS nicely zeroed before it gave it to the process. If you run even the simplest optimizer, it will say: "Well, obviously, no one cares what this variable starts with, because they didn't initialize it. So let's just use the garbage value left over in this free-at-the-moment register." Expunge your uninitialized variables, and then show me the disassembly code that the compiler got wrong. (Once upon a time I managed a piece of the validation for a C/C++/FORTRAN compiler suite. I have had this conversation more than once. :)
Exception: Real-time code with timing-specific hardware interactions. But that isn't an optimizer bug, that's a design issue.
I agree that this is quite likely. But there is more subtle evilness in undefined behavior. E.g. for "foo(bar(), baz())", it's not clear whether bar() or baz() is executed first (or maybe it is now, but wasn't in earlier iterations of the standard).
OTOH, I'm wondering if GP is building with -Wall and has eliminated all warnings? At least that's what we're doing, and this yields good results.
Yes, yes. Very true, optimizers can break in interesting ways. And yet, 99.9% of the time when some whiner says: “The optimzer broke my software!” it is an uninitialized variable.
The other 0.1% of failures kept my team busy enough. But I never allocated any time to your problem until you proved that you had no uninitialized variables.
The struggle is real! Compiling with debug enabled does change timing enough to hide (or surface!) race conditions.
The best way I've found to deal with this is:
1. Compile with -O2 (or for firmware, -Os) by default, and generate debug symbols in the elf file. Crank it up to -O3 only on a file-by-file basis with #pragma
2. Collect the process memory when an issue is hit
3. Debug it asynchronously using the debug symbols + the memory.
That way, the debugging & the running are decoupled and the timing is stable.
We're building "release with debug info" and then strip the debug info into separate files that are bundled into *-dbg packages like Debian does.
That way the backtraces can be decoded. Although sometimes they are a bit butchered because it's a release build (we use the same config while developing and it's usable).
I ran into this earlier this week. Turned out a macro that was implemented as a wrapper function in debug linked differently in ship. I don't see this first-hand very often but I'd be surprised if this kind of thing isn't an uncommon cause of ship/debug differences.
What a terrible idea. The problem which manifested as a crash before might now manifest in some number silently changing instead. And it's not like it's impossible to debug using a core dump of a "release"-compiled binary, just harder. Thousands of app developers do it every day when looking at crash reports from the field.
There are certain software bugs that cause undefined behavior. Debugging tools such as GDB can only get you so far in identifying and solving such bugs. That’s where static analysis can help.
Me: Customer is having their program X hit a SEGFAULT when they run it.
Manager: Did you transfer over a GDB version to backtrace the core dump?
Me: Yep.
Manager: What did it say?
Me: Well, when I run the same series of steps through the GDB version, it doesn't segfault, so I have nothing to backtrace.
Manager: Huh...
Me: What would you like me to do?
Manager: Move the production binary aside, drop in the GDB version. Leave it there.