
The history of peer review, and how it was troubled from the start - Hooke
http://www.nature.com/news/peer-review-troubled-from-the-start-1.19763
======
dang
This is an excellent piece about the origins and history of peer review.
Everyone interested in the ongoing controversies about replicability and bias
in science, which appear frequently on HN's front page (including now at
[https://news.ycombinator.com/item?id=11535169](https://news.ycombinator.com/item?id=11535169))
ought to read it.

------
ktRolster
The peer review system is not just to help scientists, but to help outsiders
have confidence that science is actually working (since outsiders often pay
for it), and to help them understand what consensus is. That is my summary of
the main point of the article.

I think of peer review as kind of a low-pass filter: it keeps out the obvious
junk. The problem is when science is hidden behind paywalls.

~~~
aab0
Exactly. We owe the present universality of 'peer review' to the post-WWII
expansion of government-funded military-industry-university science (see also
Vannevar Bush):

> The idea that any legitimate scientific journal ought to implement a formal
> referee system began to take hold in the decades following the Second World
> War...The very phrase 'scientific community' dates from this time.
> Researchers wanted to preserve autonomy while holding on to the massive
> government funding that had come their way since the Second World War.
> Allocations for basic research in the United States, for instance, swelled
> by a factor of 25 in less than a decade9. 'Peer review' was a term borrowed
> from the procedures that government agencies used to decide who would
> receive financial support for scientific and medical research

'Peer review', like citation counts, is essentially bureaucratic: it is the
scientific form of 'no one was ever fired for buying IBM'. ('But I relied on a
_peer-reviewed_ paper!' goes the cry. You also see this excuse often said by
authors of rubbish papers: 'you can't criticize my paper, it's been peer-
reviewed!') It is an attempt to resolve the principal-agent problem
(exacerbated by the huge increase in numbers of researchers and the prospects
of power and wealth now offered by science) by serving as a stamp of
'scientific quality' which is easily externally verifiable by ignorant
outsiders such as funders.

As such, it is considered indispensable by governments/universities, and it
helps build barriers to entry by established publishers like Elsevier. This is
why 'peer review', despite the lack of any real evidence and the occasional
study showing experimentally that it is an atrocious filter which frequently
discriminates against the best research and select for the worst, has throve
over the past century.

~~~
tamana
There is no peer reviewed evidence in favor of peer review?

~~~
mattkrause
There are, but they are fairly discouraging.

Peters and Ceci (1982) resubmitted a dozen articles after changing the
authors' names and affiliations and retyping the manuscript (this is 1982,
after all). Only 1/12 was accepted on the second go-around, while 8/12 were
rejected for quality-related reasons.

Rothwell and Martin (2000) examined data from neuroscience journals and
conferences (2 each), where reviewers grouped papers into three categories
(high/medium/low), and found virtually no agreement (they calcualted kappa, a
measure of agreement that ranges from 0 (none) to 1 (perfect), to be ~0.1).

Someone (sorry, can't find the citation; it's in the last year or two)
recently analyzed attempted to link grant scores to subsequent publications to
see if reviewers could prospectively find "good" proposals. The reviewers
could successfully identify terrible proposals and brilliant ones, but had a
terrible time with those in the vast murky middle.

------
100ideas
So much for philosopher-kings...

------
adhadh
The idea of an anonymous judge speaking for the public just doesn't make much
sense these days.

~~~
toomim
The public should be the judge.

~~~
adrianN
The public has no clue what the papers are about and doesn't really care
either.

Passing peer review is just the first step for a paper though. It also needs
to gather citations to avoid being forgotten. It's up to the scientists to
decide which paper to cite and they are usually a lot more qualified to do so.

------
methehack
Do some of the same problems apply to peer code review?

