The obsession with biases is a form of bias, you not only have to check what fake name you are submitting, you must make sure the people reading it are the same, that the mood of the people reading its the same and that their way of thinking hasn't changed much; such psychological uncertainties are the same reason psychology experiments are hard to replicate. Not to mention the randomness of things like if your resume was the first one thew saw or the last one, and many other things you have no knowledge or control over.
I completely agree that the questions that we choose to ask determine the knowledge that we get. This experiment is clearly situated within the positivist view of research.
However I don't think that invalidates the result, the whole point of randomization of people into the conditions (here the names) is to control for these latent variables like you've talked about (way of reading, mood, etc).
Are you critiquing randomization in general or this specific experiment?
Randomization solves for the problem, that's true, but that isn't what people are arguing when they criticize social science research like this.
Theoretically, if you have X number of factors you can't control for, then there should be some threshold Y where the sample size benefits from random selection and accounts for those factors. Right now, social scientists have formulas they use to establish what is and is not an acceptable population when doing experiments like this.
The problem that is being identified (and can't be stated often enough, honestly) is that human emotional preferences and variability of experience means that X is far more variable than anyone can conceive. A human being is an extremely complex biological system and that complexity compounds when you begin comparing people to one another and in groups. So unless you are actively controlling for every possible factor in that system, the assumption that you can use any formula to establish a reasonable population size Y is absurd.
Anyone with half a brain can see this, but social science has to deliver on a product and justify its existence with research funding, so it developed standards - like the aforementioned formulas - that allow it to ignore the problem of identifying X and instead just assuming their synthetic Y will do.
This is the real reason why psychologists have a replication crisis and why they always will. Many of them don't even use the standards they have because honestly, they all realize it's nonsense to begin with.
Thanks for this comment, it's well thought out and elucidates what the original poster is probably critiquing.
I want to make sure I'm understanding, are you saying that there is no sample size with which you would be comfortable making a conclusion about this? That's what I'm taking away from this comment "the assumption that you can use any formula to establish a reasonable population size Y is absurd."
What I mean is that social science research will only ever be able to speak to the samples they are studying. That's usually okay, though, because typically in an experiment with difficult variability, you can gain certainty through replication. With social science, the variability in human beings is so significant that I don't think replication works the same way. I think you would need to replicate studies about a dozen times across different cultures, timeframes, languages, regions, religions, etc just to even begin approaching something like reliable results.
And given the fact that social science has a replication problem (an understatement if I'm right), the entire area of study is suspect.
Unfortunately there is no clear answer to that, I'm latino but living and working in a south american country so its irrelevant; but I have searched for jobs overseas (remote) and a few times I have landed the job.
