What kind of tooling were you using? IMO, deadlocks are some of the easiest concurrency bugs to diagnose. If you can see the thread stacks, it is easy to see threads are blocked from acquiring a lock. If you can attach a debugger, it is easy to see which locks are involved. Then you can pretty much figure things out using the straightforward guideline that if multiple locks are involved, they must be acquired in the same order in all code paths.
I don't want to devalue your experience, but I am surprised to hear that. Livelock is harder to debug. Silent data corruption caused by missing or wrong synchronization is way harder to debug.
Agree completely. But they become harder when you eschew standard constructs like threads and mutexes and bring-your-own losing nice things like debugger supports and stack traces. Now your logical tasks might be deadlocked, while your threads appear to be running correctly. This is surprisingly common this day with async runtimes and less than stellar debugging support.
> Silent data corruption caused by missing or wrong synchronization is way harder to debug.
Reminds me of being a young and ambitious C++ programmer 25 years ago, discovering that when you have a map and do “return m[k]”, it is not, in fact, a read-only operation when k does not exist. After which I learned that const-correctness is not just a nice-to-have, especially in multithreaded applications.
But yeah deadlocks are hardly ever a difficult issue to diagnose. They may potentially be difficult to resolve, but at that point, it very much suggests that there’s an architecture / design issue.
Just drain the traffic (make the load balancer temporarily not send traffic to it) and attach a debugger. I hope your load balancer has that feature because health checking also needs it.
ah, that's probably a missed wakeup problem! Much nastier. Still seeing where your thread is blocked might hint you to where the missing signal should have been.
So I actually had this exact issue where we suspected a deadlock, but the application was limping along and we couldn't justify attaching a debugger. I managed to confirm my suspicions with judicious use of perf and /proc/<pid>/task/<tid>/wchan .
