I've been doing something similar for the past few years. I have a development machine running tmux + vim that I can connect to via ssh / mosh. Biggest issue I've run into was high latency while in Europe (since my box is in the US), but otherwise it works great, even on flaky connections. The ability to download / build large docker images with a beefy computer is a very nice feature as you mentioned.
