So basically, social sciences are still at the sad state of affairs Feynman described.
From the article:
>"One reason so many of these Psychological Science studies are so dead on arrival is that they hinge on noisy measurements in uncontrolled, between-subject designs. That puts you right here, and no amount of preregistration or fancy statistics is going to solve your problems."
It must be great to work in a field where you decide what the result is. You can't do that in mathematics unfortunately. It must be why all these mathematicians are so bitter with social sciences.
I think Feynman was maybe a bit too harsh, possibly because he was so used to the relatively pristine world that physicists get to live in that he outright balked at the noise psychologists and social scientists have to live with.
You do get noisy measurements. That's why you need to pay a lot of attention to your confidence intervals and standard errors - they give you a sense of your signal to noise ratio. Not all psychologists and social scientists are good at that, and that's a problem, but IMO it's really not _that_ difficult to distinguish babies and bathwater here.
And I think Feynman's comment about between-subject designs is outright dangerous. You do not necessarily reduce your noise by switching to within-subject designs. You just move your noise to a place where it's much more difficult to measure, and possibly create a lot of endogeneity that you can't deal with except by throwing out a bunch of assumptions and hoping you're right. Sure the numbers that spit out of your equations start looking more tidy, but that doesn't mean your results are more trustworthy.
Oops, the quote was from the article, not Feynman (I've edited my post now); note that the author is Gelman from 'the statistical crisis in science'.
But it throws back to Feynman's speeches on Cargo cult science, what is science and what isn't.
The idea is basically the same, we can describe what good science is by the rigor put in the experimental process, but this is not something that can be captured simply by the p-value.
Simpson's paradox should give pause to anyone trying to run statistics on data from poorly selected test subjects. Not only can you get results that are a mirage, you can get results that are the opposite of reality.
When it's well intentioned... A friend went to a statistics-focused master degree, and her first internship made her come back with the comment "well, during class we had one prof tell us exactly what we shouldn't do with the data to analyze it honestly. That's the exact list of tasks I was given for these last few months."
I'm sort of reminded of a comment by an ex who grew up in a poor neighborhood. The cops learn how to fudge their reports so they get past the judges. Replace court judges with peer reviewers.
From the article:
>"One reason so many of these Psychological Science studies are so dead on arrival is that they hinge on noisy measurements in uncontrolled, between-subject designs. That puts you right here, and no amount of preregistration or fancy statistics is going to solve your problems."
It must be great to work in a field where you decide what the result is. You can't do that in mathematics unfortunately. It must be why all these mathematicians are so bitter with social sciences.