This isn't mysterious or hard, and I've written comments about it before (https://news.ycombinator.com/item?id=7073079). There are two ways to lower the cost of something: increase the supply or decrease demand. It's quite easy to build more units on a given parcel of land but SF has mostly chosen not to do so (http://www.theatlanticcities.com/housing/2013/10/san-francis...).
The real problem starts with voters, since they're the ones electing the mayor, city council, and other officials, who respond to voter preferences against building stuff but who also deal with voters complaining about high rents. They're in a situation similar to the ones Bryan Caplan describes in The Myth of the Rational Voter, which is a fascinating book that deserves not just to be read but reread (http://www.amazon.com/The-Myth-Rational-Voter-Democracies/dp...).
More generally, if you get more supply in the face of increasing demand, but not enough to lead to actual decreases, you still get smaller increases, per basic micro econ.
About Houston: is Iglesias the source for that? If not, where would I read more? and is the buildout preceeding reduced prices post 2008?