I don't think the issue is so much about skin color, as it is about language proficiency and integration. If you aren't fluent in Swedish or very proficient in western-style English, you have a significant uphill battle in terms of career and life prospects.
You have Iranians and Syrians and what have you in all manner of well paying jobs. But there is really no plausible path from refugee to those types of jobs.
I wasn't saying a racial glass ceiling but a ethnic one, largely for the same reasons you mentioned. Lack of language proficiency plus a bit of bias against foreign degrees forced plenty of white collar immigrants to take blue collar jobs in Europe (and even Canada). This decreases social mobility as you don't have the network to get an "in" into plenty of well paying industries. While skills absolutely matter, some form of network really helps out.
You have Iranians and Syrians and what have you in all manner of well paying jobs. But there is really no plausible path from refugee to those types of jobs.