For California specifically, there's proposition 13 which caps the rate of increase of property taxes for housing to a rate that's below inflation.
For areas like the Bay area where the cost of living is high (making city employees expensive), zoning for residential instead of commercial means that they're limiting their future tax base while having to provide services to these houses, so cities avoid building housing as much as possible.
Prop 13 restricts reassessments for commercial properties, too. Ultimately, the fundamental calculus is similar in that commercial development is deemed to be better for city finances, but the difference is much smaller than if commercial properties weren't part of Prop 13.[1] And perhaps the difference disappears or inverts in urban contexts. Suburban services aren't financially viable long-term even for commercial zones, but that's not true in dense, urban areas where inviting people to live can definitely have a positive RoI long-term.
The reason urban areas, especially coastal cities, don't permit more housing has more to do with NIMBYism and, more recently, leftist politics that sees all private and even much public development as evil--because so-called gentrification, because money changes hands, etc.
[1] Note that the way schools are organized and funded in California means there's less local disincentive stemming from the need to fund schools than might exist elsewhere. I'm not familiar with the numbers, but if you're a municipality looking to keep tax money flowing through your area as opposed to being redistributed elsewhere, I wouldn't be surprised if public schools are a net win.
For areas like the Bay area where the cost of living is high (making city employees expensive), zoning for residential instead of commercial means that they're limiting their future tax base while having to provide services to these houses, so cities avoid building housing as much as possible.