Please could you explain how ASLR makes your programs impossible to debug?
I'm surprised to hear this, as I work on such executables in my debugger every day, with no problems whatsoever caused by them having ASLR enabled.
> On the other hand, stuff like ASAN is great.
It is, but it's not intended as an exploit mitigation. You typically wouldn't apply it to your binaries running in production, due to the performance hit. Its purpose is to help detect memory safety bugs during development and testing.