I did a few times. It's actually easier to make something look exactly the way you want it to look in plain TeX than fight with some poorly documented LaTeX package.
I'm sure its possible and useful for specialty cases but would you want to write TAOCP in it?
My experience of using LaTeX compared to WYSIWYG is that it's easier to focus on the text in the editor instead of fighting with the UI interface but it then requires some chasing around package bugs and compile errors. In total it's clearly a net positive for me.
Writing pure TeX would seem to push the tradeoff quite a bit, having so much distracting stuff around the text that requires a totally different mindset than the text being written. It was probably designed for a world where typesetting was done as a final step before publication, potentially by someone else.
The idea is that you put together the few macros that you need and then the book is mostly tag-free.
IDK you can find the source to the TeXbook on CTAN, see if it seems too noisy to you (I don't remember).
