There's an interesting idea... though I have no idea how well it would work. The bandwidth is still massively below what graphics cards typically need. With some OS changes and essentially half-remote-desktopping to a dumb terminal display, it could work, though...
You have given me thoughts. Thanks :D