In almost all countries (including non-White countries) "immigrants" when used as a pejorative is code-word for race.
When Americans complain about "immigrants", they sure as shit don't mean Gunther from Germany, they mean Juan from Mexico or Chang from Korea. Europe is not immune from this - France as a whole seems much more staunchly racist (or rather, accepting of casual racism) than the USA.
Racism has many faces, especially today where it's now taboo. There are all kinds of ways that racism manifests in modern, developed society without someone screaming "he's black! get him!".
