I've been interested in a possible analysis for a while (maybe I should finally dig in and do it myself). My hypothesis is that San Francisco isn't as nimby-istic as it appears, that this is an artifact of a political and geographical oddity. San Francisco is a 7x7 square mile city and county that is already reasonably dense (on average - parts are extremely dense, the only place outside NYC with 100k+/sq mi density, whereas other neighborhoods 4 miles away are leafy suburbs). It's also a peninsula, and much of the area to the south is designated open space.
Here's the thing - I'd be interested in analyzing other metro regions where a long standing, dense urban core isn't its own city and county, and see what percentage of growth happens within it or outside it. In other words, let's look at sub-regions with existing stable population density of above (say, 20,000 inhabitants per square mile for the last 50+ years) and see how much construction has occurred within them.
For instance, does Atlanta have a small, historic downtown with an existing housing base that has been relatively dense and stable for 50 years? Has the growth in Atlanta come from tearing down 80 year old buildings in neighborhoods with 20k/sq mile populations dating back a while, or has it been in the suburbs and exurbs that count as "Atlanta" but wouldn't count as "San Francisco"?
I don't know the answer to this question, but I think it's an important one to answer before concluding that SF is uniquely "nimby-istic" toward housing in some cultural or political way. You can probably tell where I'm leaning with this, though - that the NIMBY attitude, to the extent that it exists, has more to do with low density areas like mountain view resisting new housing than high density areas like much of SF.
If San Francisco were more NIMBY-tastic than its sister California coastal cities of Los Angeles and San Diego, then one might expect its housing prices to pull away from those two.
Prices in SF, LA, SD all move together with the economy but increases in SF have outpaces almost every other city. This has been going on for decades and hard to see on your chart as differences in price changes are compounded up over the years. 50 years ago prices were much closer together today they are far apart:
This doesn't address the larger region, though. Some of the development on the outskirts of San Diego counts as San Diego, whereas anything outside the 7x7 grid of San Francisco wouldn't show up in the numbers.
Take some of San Diego's more historic neighborhoods that were build a long time ago, and track the home value. For instance, how much has a 4 bedroom 3 bath spanish style house build in the 1920s near La Jolla Cove (or in Mission Hills, if you want to keep it in SD proper) changed in value? That would be a better basis of comparison to SF.
I agree that without getting into the details you can be misled. "Cities" are defined many ways: city, county (same as city in SF but not in LA or SD), metropolitan division, metropolitan statistical area and combined metropolitan statistical area depending on your data source. Frankly it is a big mess and often you are not told which definition is being used. Not only that the definitions change over time! I had to deal with this data for many years - not simple at all.
Which one is most appropriate depends on your goal. La Jolla or Mission Hills are probably a lot more like SF than the SD MSA so likely perform more like SF but will still be tied to SD.
Unfortunately sales only occur infrequently and tend to dry up completely in recessions. This along with lack of data as well as different definitions make tracking detailed property types like 4br 3ba Spanish in La Jolla difficult and unreliable if not impossible. And in SF (city & county) that property type is very rare (also SF does not publish style data like some area - I think SD might). Best you can usuallydo is by bedrooms in a zip but only if the market is pretty active.
My original point was a simple one: even though real estate moves together across the country with the business cycle over time area can still diverge significantly. This has happened in SF vs LA/SD and in California as a whole vs the rest of the US. To see this look at the index levels over longer periods. Maybe go with a big areas like MSA/CMSA so you have lots of data and get both mature inner city and fringe suburbs - this is what Case-Shiller public indexes are like.
Here's the thing - I'd be interested in analyzing other metro regions where a long standing, dense urban core isn't its own city and county, and see what percentage of growth happens within it or outside it. In other words, let's look at sub-regions with existing stable population density of above (say, 20,000 inhabitants per square mile for the last 50+ years) and see how much construction has occurred within them.
For instance, does Atlanta have a small, historic downtown with an existing housing base that has been relatively dense and stable for 50 years? Has the growth in Atlanta come from tearing down 80 year old buildings in neighborhoods with 20k/sq mile populations dating back a while, or has it been in the suburbs and exurbs that count as "Atlanta" but wouldn't count as "San Francisco"?
I don't know the answer to this question, but I think it's an important one to answer before concluding that SF is uniquely "nimby-istic" toward housing in some cultural or political way. You can probably tell where I'm leaning with this, though - that the NIMBY attitude, to the extent that it exists, has more to do with low density areas like mountain view resisting new housing than high density areas like much of SF.