This is pretty close to how steno [https://www.openstenoproject.org/] works in practice and worth learning IMO if you are going to go this far. In fact I have a few emacs-specific chords, but not as many as you might think. The whole experience is a bit like if emacs is now your keyboard, so it happens to go great with emacs.
[1]