
Physics tests with a female bio get worse grades from inexperienced teachers - AdmiralAsshat
http://arstechnica.com/science/2016/01/give-teachers-a-physics-test-from-a-girl-and-theyll-give-it-worse-grades/
======
cameldrv
Looks like a null result that has been p-value hacked. Without the (paywalled)
original paper it's tougher to say, but:

The effect only occurs in teachers with less than 10 years experience, and
only teachers in Switzerland and Austria, not in Germany. Population of
Germany: 80 million. Population of Austria: 8 million. German speaking
population of Switzerland: 5 million. Approximate average career length for a
Physics teacher in German speaking countries: 30 years.

So about 4.5% of their sample exhibited a bias against girls. The typical
p-value to accept a result is .05. If we sliced and diced the sample into 20
groups, we would expect one to show a significant effect at p<.05, which is
exactly what they showed. Note that they did a retrospective analysis based on
years of teaching experience, gender of teacher, country of teacher, and fake
major of student. There are plenty of combinations of subgroups to find
something that makes it over the p<.05 hurdle.

Can't find a non paywalled full text of the original paper, but it looks
strongly like p-value hacking a null result to me. They found no effect in 80
million population Germany, and only an effect among teachers with less than
10 years of experience in 8 million population Austria and 5 million
population German-speaking Switzerland. If we assume that their sample is
evenly distributed across German speaking Europe (The test question was in
German),

------
ck2
The one-time generic ID thing is an excellent idea.

In fact they should have run a control like that.

