I think this actually gets to the heart of one of the big problems with this study. When you ask the model to impersonate a Republican or Democrat and then ask some questions it's worth thinking about what training would inform the answers to those questions. A lot of the training would surely be hot takes from the internet which straw man the position of the other side and present a more extreme view. This is because when people say something they believe they are going to most often say what they think ("I think x"), not as often "I am a republican and I think X". whereas if they are straw manning the other side's political pov they will absolutely say "Republicans all think x".
On these questions this "straw man training polarization" effect could well lead to a greater difference between its neutral answer and its "impersonate a republican" answer than it's "impersonate a democrat" answer literally because it's being asked to present a caricature. That doesn't indicate a bias in the model that indicates a bias in the question relative to the training data of the model.
Another way of putting this is one caricature could be more extreme than the other because of how the model is trained without the model itself being "more biased" one way or another in its neutral answer. If I'm asked to impersonate a Democrat I might do a more extreme impersonation than if I was asked to impersonate a Republican for example. That doesn't mean I'm more of a Republican than a Democrat. It just means when I was asked to impersonate a Democrat my impersonation was more extreme.
This is a very significant methodological flaw in my opinion. I notice the author has no background in stats, social sciences or AI. He's an accounting PhD[1]. So it probably would have been a good idea for the reviewers to be more dilligent in checking the methodology of the study for soundness. Here's the paper btw https://link.springer.com/article/10.1007/s11127-023-01097-2
1. They don't establish correlation of the ChatGPT impersonations of Republicans and Democrats with the real views of actual Republicans and Democrats. They should have had a representative sample of each actually do the compass test and compared the distributions of those with the distributions of the impersonation results to ensure they are calibrated. As such they are at best showing that on this test, ChatGPT's answers are more like the answers chatgpt gives when it impersonates a Democrat and less like when chatgpt impersonates a Republican. This doesn't say anything at all about true bias, just about how chatgpt impersonates groups.
2. Say chatgpt's impersonations are calibrated to real democrat and republican views (ie ignore 1). They seem to assume non-bias means equidistance from the republican and democrat positions in lots of important ways. eg "If ChatGPT is non-biased, we would expect that the answers from its default do not align neither with the Democrat nor the Republican impersonation, meaning that 𝛽1 = 0 for any impersonation" well, no. Bias isn't a function of what Democrats or Republicans think, bias is lack of actual neutrality in some sort of Platonic sense. Ie if this questionnaire was 100% calibrated then neutrality would be the origin of the coordinates. Given that, say democrats currently have a default position that is more extreme in one or other (or all) dimensions on this questionnaire than Republicans, then a neutral position would be closer to the Republican position and vice versa. If uncomfortable with the abstract definition of neutrality here then maybe a better one would be to pick a third representative sample of politically neutral people and calibrate their views relative to the test. Then neutrality or bias would be distance from this group not distance from the centroid between Democrats and Republicans.
3. These problems (1&2) are repeated in the "Professions" section, and compounded by a slightly weird inference about referencing the population mean (Republican/Democrat) of each group without calibrating any of the actual responses with any actual real person's responses (ie still just comparing chatgpt to itself impersonating someone).
4. They say they checked the Political Compass test versus IDR Labs Political Coordinates test "as a robustness test", but don't show any results from IDR labs or comparison between the result sets. That seems very odd.
I personally think this topic is very important so I find this all in all to be an extremely disappointing paper, which will nevertheless no doubt garner significant press attention without people actually reading it or trying to understand the methodology or conclusions.
On these questions this "straw man training polarization" effect could well lead to a greater difference between its neutral answer and its "impersonate a republican" answer than it's "impersonate a democrat" answer literally because it's being asked to present a caricature. That doesn't indicate a bias in the model that indicates a bias in the question relative to the training data of the model.
Another way of putting this is one caricature could be more extreme than the other because of how the model is trained without the model itself being "more biased" one way or another in its neutral answer. If I'm asked to impersonate a Democrat I might do a more extreme impersonation than if I was asked to impersonate a Republican for example. That doesn't mean I'm more of a Republican than a Democrat. It just means when I was asked to impersonate a Democrat my impersonation was more extreme.
This is a very significant methodological flaw in my opinion. I notice the author has no background in stats, social sciences or AI. He's an accounting PhD[1]. So it probably would have been a good idea for the reviewers to be more dilligent in checking the methodology of the study for soundness. Here's the paper btw https://link.springer.com/article/10.1007/s11127-023-01097-2
[1] https://research-portal.uea.ac.uk/en/persons/fabio-motoki