For the same reason that people can afford food even though a minority is able to buy all the food.Just because a minority can buy something doesn't mean they will. If it gets easy enough to build more housing then it won't make sense to buy houses as an investment; the threat of oversupply would be too great.
FWIW, unrelated to this topic, I think we should adopt Georgist tax policies. That has nothing to do with inequality and it would resolve the situation of wealthy people buying property neatly.
Thats a really bad comparison since food is a consumable and most time not a investment since its expires (with exceptions like whisky).
Housing is a good example since our planet has a natural limit which can't be worked around. Otherwise please tell me why Bill Gates owns about 270,000 acres of farmland. There is a famous quote about it. “Buy land—they aren’t making it anymore.”
If you got spare money you are pressured to invest it, since the inflation will reduce the value. But not everyone is able. Thats where your financial gains are coming through you investments. By people who are not able to invest themself and the cake is getting smaller every year.
FWIW, unrelated to this topic, I think we should adopt Georgist tax policies. That has nothing to do with inequality and it would resolve the situation of wealthy people buying property neatly.