As someone living in Tokyo. This xenophobic country is glorious. The problem is largely overblown and people are beyond nice. Easy to get friends and to date. Only problem is the working hours, otherwise it's way better than most countries. Only other place that I like as much is Sydney. Protip: be thin and behave well (no macho attitude) and you'll do well.
I would not say you're wrong but I believe it's important to note that the racism is not blatant, but rather perverse. As an example, foreigners working in a large Japanese organization will often feel a clear glass ceiling blocking their career advancement, regardless of their results, because of not being Japanese.
Another form of xenophobia is with the rental market, where a large proportion of places is simply out of reach for foreigners, regardless of their income level. Positive point is that this is usually not the case for the more premium bracket.
On the other hand, immigrating to Japan is very simple for qualified labor, which is great for foreigners looking for some experience here.
That's more likely to do with that foreigners will not know perfect Japanese so you'll reflect badly on the company. It's not racism lurking. Especially since the language has so many levels of politeness that means that even bending of the words are different depending on who you talk to. Business Japanese is very difficult even for Japanese. But it's always easier to hide behind anger. Just realize that many of the things that make Japan great has a drawback to them. But I prefer that countries are different so you can find a place that suits you.