
How Did That Make It Through Peer Review? - iamjeff
http://blogs.plos.org/paleocomm/2016/02/03/how-did-that-make-it-through-peer-review/
======
dekhn
Journal of Cardiology maintains a full-time statistics editor who reviews
every single paper before publication. They routinely catch basic stats errors
and send the papers back with a number of repairs required for the paper to be
published.

That Nature and Science and other prominent journals continue to publish
statistically invalid papers without verifying them (and this isn't even peer
review, it's just technical validation) amazes me.

------
anbende
The author tries to make a case that this is not about incompetence.

One of the primary examples given is the case where shoddy analyses sail
through peer review, because none of the reviewers know enough about those
analyses to comment.

It's hard for me not to see this as incompetent. Are the reviewers incompetent
for not knowing this particular analysis well enough to review it? Often! Most
scientists in my experience (graduate student) have a pretty bad understanding
of statistics and perform those analyses that will "pass peer review" rather
than those which come from a comprehensive knowledge of the topic. And even if
the analysis is new or uncommon, giving the paper a pass without mentioning
that the analysis is beyond them is pretty problematic but is sadly also the
norm (usually they could learn enough to review it in a few hours if they
really wanted to!).

Then there's the more general fact that no one reviewing the paper is
qualified to comment on what is arguably the most critical part - whether
analyses actually show what the authors think they do! If no one involved
knows how to do the analyses in question then the team is not competent to
perform or review this work.

That's the result of sometimes an incompetent editor and usually a horribly
broken system. The fact that checking the numbers isn't of primary importance
in reviewing a lot of scientific work is a travesty.

------
chris_wot
Makes you wonder whether peer review is working if a true subject matter
expert doesn't do the review. Sounds like pointless rubber stamping and
checkbox ticking.

