Debugging software in production is a hard unsolved problem, the available tools are pretty primitive, and it doesn't help that people often don't plan it into their development and deployment cycle.
We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging.
I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).
In languages like C it's an unsolved problem. Doesn't mean it's an unsolved problem everywhere (see, e.g. Erlang, which even allows for hot patching without even taking the system down).
We had a client once using our system, who would see anomalies only after long uptimes at sustained high loads. It took six months to find the cause (a race condition in the database code) because we had so few tools at our disposal to look at the post-mortem state of the program. Running the software at those loads for the weeks or months it took to reproduce the problem under a debugger was obviously not a tenable proposition, and reproducing the variability of the particular real-world traffic was difficult to do in a controlled environment. And the software wasn't written with any eye towards making state available for post-mortem debugging.
I guess "don't write complicated multi-threaded software in C" is a pretty good response to the above too. That particular bug couldn't have happened in say Rust (though it could have in Java).