The Phrack "vudo" writeup from 2001 was about getting arbitrary code execution given a one-byte out-of-bounds heap write that replaced something with a NUL byte and immediately set it back.
As far as compression algorithms go, DNS' algorithm is ridiculously simple. Switching compression algorithms wouldn't improve security. At best it'd be a lateral move; more likely it'd be a net security loss. We would likely get better compression, though.
If the response is that people would just reuse pre-existing compression libraries, the exact same thing could be said here. In fact, Unix systems provide `dn_expand` and `dn_comp` in libc (e.g. BSDs, Linux/glibc, Linux/musl) or another system library (e.g. -lresolv on Solaris). And of course nginx could have used one of countless other DNS libraries, such as libunbound.
DoH has lots of uptake. So does DKIM. The reason DNSSEC is moribund is that it doesn't solve real problems for ordinary people. I wouldn't be so pessimistic about getting real new stuff deployed.
What's insane on DNS encoding scheme? Do you consider binary encoding insane? Would you like to re-engineer protocols down to Ethernet to use JSON? For what reason exactly?
You suggest breaking compatibility for everyone on this planet without giving any specific reason for doing so.
Please read the rfc, then think about how a naive implementation of the name encoding can be abused.
We have had many vulnerabilities related to that single section of the standard over 30 years. At some point you have to ask yourself if the problem lies with the developers or the standard.
The patch described on the vulnerability page worked 1-to-1 for in two nginx versions I fixed. But in nginx 1.20 in ./src/core/ngx_resolver.c the second replacement does not match the existing if (len == -1){contents} and I had to remove some extra bits based on what seemed right it manually.
http://phrack.org/issues/57/8.html