There are C compilers out there that work like this (like, say, the Plan 9 C compiler) and they rarely ever get used in practice, because GCC and clang -O0 are fast enough. I think making a second compiler backend just to eliminate the IR generation step isn't going to be the solution we want in the long run.
I think they are only fast enough because developers have not been exposed to better. In general, I would like to see more aggressive build modes for CI servers, and less aggressive modes for dev.