

Theory-testing in psychology and physics: a methodological paradox (1967) [pdf] - gwern
http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf

======
capnrefsmmat
My favorite quote from this has to be

 _Meanwhile our eager-beaver researcher, undismayed by logic-of-science
considerations and relying blissfully on the “exactitude” of modern
statistical hypothesis-testing, has produced a long publication list and been
promoted to a full professorship. In terms of his contribution to the enduring
body of psychological knowledge, he has done hardly anything. His true
position is that of a potent-but-sterile intellectual rake, who leaves in his
merry path a long train of ravished maidens but no viable scientific
offspring._

The problem boils down to the insistence on statistical significance testing.
Whenever you test an intervention (priming experiments or whatever a
psychologist might look for), it's reasonable to expect there's _some_ effect
-- even if it's negligibly small or practically useless. But significance
testing is about proving that the effect is not zero, so you will _always_ get
a significant result as long as you collect enough data.

This is why I am so frustrated when I see a psychology paper claiming
something like "Women wear pink shirts while more fertile" and the paper says
the effect is statistically significant, but _doesn 't_ say how _big_ it is.
Maybe it's a completely unimportant effect.

This is why someone (I forget who) suggested that null hypothesis significance
testing (NHST) should be renamed Statistical Hypothesis Inference Testing, for
the aptness of the resulting acronym.

There's a whole legion of other common problems (enough to write a book on
[see my profile]), but in psychology especially this is a big one.

------
misnome
Probably also worth a read is Feynman's Caltech commencement address, which
talks a little about Psychology (or the state thereof in the 70's - I don't
know enough to judge whether things have really changed, and doubt the media
provides a balanced perspective)

[http://neurotheory.columbia.edu/~ken/cargo_cult.html](http://neurotheory.columbia.edu/~ken/cargo_cult.html)

------
gwern
Excerpts of Meehl 1967:
[https://plus.google.com/103530621949492999968/posts/4sSbzwnP...](https://plus.google.com/103530621949492999968/posts/4sSbzwnPkwd)

