If 5 people can afford the real estate and 15 people cannot, it isn't that weird that 5 people are staying and 15 are leaving. That the price equilibrium is reached means that a lot of people can't afford those prices, which will result in a net loss of population until a population equilibrium is reached. And it isn't weird for very expensive but smaller places like Aspen to have already reached both of those equilibriums, so their population is relatively stable but they are still very expensive places to live.
20 people arrived in town, 5 found housing that they could sustain, 15 did not. Unfortunately, since residents who are renting bid for the same housing as newcomers, they can get pushed out as well (5 residents with housing, 15 new comers, not all the stayers will be existing residents).
