It is elected officials that make these decisions, and until people with your point of view start running for office and getting elected things won't change. As of right now it looks like the majority, however slim, of the voters in SFBA are not choosing to change. At least not as quickly as the problem seems to be progressing.
The problem is that the people impacted by the problem wind up with long commutes, and therefore do not live in the places whose policies would have to change.
If you could choose to vote where you work instead of where you live, this problem would go away very quickly.
NIMBY is actually a very vocal minority. A lot of them are retired folks who have a lot of time to engage in City and County politics. I do a little bit and write to City, State, and County officials. But between a long commute and a kid and a dog, I just don't have time to dedicate on this. And that's the problem with local politics.