
“Psychological Science” backs away from null hypothesis significance testing - tokenadult
http://andrewgelman.com/2016/03/02/no-this-post-is-not-30-days-early-psychological-science-backs-away-from-null-hypothesis-significance-testing/
======
logicrook
So basically, social sciences are still at the sad state of affairs Feynman
described.

From the article:

>"One reason so many of these Psychological Science studies are so dead on
arrival is that they hinge on noisy measurements in uncontrolled, between-
subject designs. That puts you right here, and no amount of preregistration or
fancy statistics is going to solve your problems."

It must be great to work in a field where you decide what the result is. You
can't do that in mathematics unfortunately. It must be why all these
mathematicians are so bitter with social sciences.

~~~
bunderbunder
I think Feynman was maybe a bit too harsh, possibly because he was so used to
the relatively pristine world that physicists get to live in that he outright
balked at the noise psychologists and social scientists have to live with.

You do get noisy measurements. That's why you need to pay a lot of attention
to your confidence intervals and standard errors - they give you a sense of
your signal to noise ratio. Not all psychologists and social scientists are
good at that, and that's a problem, but IMO it's really not _that_ difficult
to distinguish babies and bathwater here.

And I think Feynman's comment about between-subject designs is outright
dangerous. You do not necessarily reduce your noise by switching to within-
subject designs. You just move your noise to a place where it's much more
difficult to measure, and possibly create a lot of endogeneity that you can't
deal with except by throwing out a bunch of assumptions and hoping you're
right. Sure the numbers that spit out of your equations start looking more
tidy, but that doesn't mean your results are more trustworthy.

~~~
logicrook
Oops, the quote was from the article, not Feynman (I've edited my post now);
note that the author is Gelman from 'the statistical crisis in science'.

But it throws back to Feynman's speeches on Cargo cult science, what is
science and what isn't. The idea is basically the same, we can describe what
good science is by the rigor put in the experimental process, but this is not
something that can be captured simply by the p-value.

~~~
euyyn
FWIW, your new wording still gave me the impression it was a quote from
Feynman. Just one that you copied from the article :)

~~~
kevinwang
That's what I thought as well.

------
lr4444lr
This:

>"All too often I feel like I’m seeing the attitude that statistical
significance is a win or a proof of correctness, and I think this pushes
researchers in the direction of going the cheap route, rolling the dice, and
hoping for a low p-value that can be published. But when measurements are
biased, noisy, and poorly controlled, even if you happen to get that p less
than .05, it won’t really be telling you anything."

~~~
Gibbon1
About a year ago, saw youtube video about p values. Can't find it again. The
upshot was p-values are really noisy. And I think mentioned p-values tend to
work the opposite of the way they are used. Greater than 0.05 most likely
means 'nope'. Less than 0.05 means 'maybe'

