I once did an 'always on' google hangout with another guy when I was working on a software project with him. It was remarkably effective substitute for being in the same room. Key was it was always on and we could speak to each other when necessary. Nobody does this although in the companies I have worked in.
Also many meeting spaces have microphone quality issues, and it can be hard to hear people in the other meeting room as they have conversations with each other.
Culturally I think it will be hard to pull off, virtue of video conferencing not being used in the same way today.
I'm always surprised by the fact that more companies don't adopt the "always on" google hangout. It's literally in the name of the service, "hangout" yet people use it like a traditional conferencing tool.
If more people adopted it as a ROOM that you all worked in, I have a feeling many of the remote nay sayers would at least feel a little bit better about things.