That's only a small part of it. Most of the people are the richest in their extended family, and tons of friends from HS, college, etc. so they do know people who are not in their income bracket.
I'd say it's more related to the imposter syndrome. IE if I got here and I'm just ok, clearly this wasn't too hard a level to reach.
That and "well I used to work with so-and-so, and we're about equal, but they're making $800k while I'm stuck making $400k, so I'm clearly failing". Or even $2M vs $10M, etc.
Again, it's crazy, because both people are clearly actually "rich", I'm just trying to explain the mechanics of why so few actually feel that way.