But they compete as governments, rather than as direct builders. That makes it really hard for it to work as a market.
Cities can and often do act as independant (global) agents (bidding for Olympic games comes to mind, and offering special sweetheart deals to banks etc). London has a revenue and expenditure of roughly 22BN - putting it around 150th in the Fortune 500, and it is far mnore concerned with attracting and retaining talent from New York or Paris than from Sheffield.
If London fails to provide the right mix of house building and amenities, those people and businesses will (eventually) go elsewhere.
Humans do not physically move often or easily - but cities do rise and fall and compete with each other - they are the right level IMO to look at "good neighbourhoods" and if we want to solve the "livability" of cities, then cities (or possibly sub-city-areas) are the right sized unit for the discussion.
Its not about houses, its about neighbourhoods.
There is a problem of indirection, as chez17 alludes to. A "city" is too much: a place to do business, a place to live alone, a place to explore, a place to retire, and so on and so on. Some of these measures are so critical that failure can be crippling. Thus, a city might compete for the attention of international organizations like banks and the Olympic committee... but why? What's the actual value exchange happening there and how does it play out in the details? (I'm not asking "why should they", but "why are they".)
In answer to why - tax revenue is one less than compelling answer - my preference is to look back at the city states of Greece - they competed and fought for supremacy, for survival for riches - but mostly they fought because their city was their tribe and their home and the others were enemy. It's pretty human basic stuff I believe - nations are too big to get really worked up about but a city - it's the right sort of size.