>categorizing people by skin colour is ridiculous for almost all purposes
Sure. But race is most often about genetic (or ethno-linguistic) heritage, not skin color. A person from Japan might have the same skin color as someone from Greece.
Only three large racial (genetic) groups in Sub-Saharan Africa.
>Which one exactly is the genetically stupid one?
You're trying to turn this into a normative claim, which it is not. And you started with the assumption that people are just saying this stuff for no reason at all.
Also, didn't "black" and "n-word" switch meanings as slur/non-slur less than a century ago (and might switch them again in less than a century) ?