The flip side of this is that a large number of Jewish people in Germany maintained their clear cultural separation for around 500 years even though the 'looking different' aspect seems to have diminished over time. In retrospect you'd have to say that outperforming the locals without becoming one of them isn't appreciated.
You mean: "... if you don't look white or black..." Just ask anyone to draw a picture of an "All-American". Most folks will draw someone white or black.