Improve it all you want, I'm totally into that. The problem is that once your "terminal" ceases to be a terminal emulator then you lose so much it's pretty unrecoverable.
Most of the things that you suggest are very much technically possible for a shell (perhaps zsh with some extensions) running in xterm. You'd be left with some sort of unholy combination of a rather modernized `mc` and an otherwise modern shell, but it'd be a very interesting product.
tl;dr: We're not opposed to improving the 'terminal'. I'm opposed to 1) mistaking "improving the terminal" with "improving the shell" and 2) losing the terminal.
If you do not provide direct user access to a pseudo terminal through a terminal emulator, then you will undoubtedly lose functionality. You need both the pseudo-terminal and the terminal emulator. If you do provide direct user access to the pty through a terminal emulator, then what the hell is the rest of the crap there for?