We've considered moving, sure. For us though the cost of moving houses in this market, and my wife's local reputation-based business (wedding photographer) would make it an extremely expensive proposition in the short term. I think getting left behind is definitely a risk. So far I'm the only employee outside of the bay, but the culture has remained pretty asynchronous and conducive. If we continue to find talent local to SF I imagine there's a solid chance that'll change, if we happen upon more awesome remote talent it might not. I think it'll just depend on who we end up hiring over time.
Early on though, getting big enough to have that issue sounded like a very high class problem so we decided not to care much where people were and focus on getting things built.
[Aside: That's awesome that you're in SD! Drop me a line and I'd be happy to pass your details along to some of my SD developer friends]
We're also in SD, but trying to build a team here and not finding as many local applicants as we'd like.