I dont think anyone would care if some people just had huge piles of money in the bank. Its what they do with the money that causes problems. That's income.
This countries lack of affordable housing is due to NIMBYs pulling the ladder up after themselves. We could build enough housing for all the short term rentals and second homes people want if it was legal. Rich people owning houses isnt a problem unless theyre leaving them vacant en masse which by and large is not happening.
Its not the entire world, mostly the US. Japan for example doesnt have a NIMBY problem and that has allowed tokyo to become incredibly dense without becoming expensive.
Where do you live? Even manhattan has vacancy rates barely above the 3% structural minimum and thats the poster child for "rich people leaving property vacant" complaints.
The problem really is that too many homeowners refuse to allow anything other than SFHs to be built near them.