
We shouldn’t take peer review as the ‘gold standard’ - danso
https://www.washingtonpost.com/outlook/why-we-shouldnt-take-peer-review-as-the-gold-standard/2019/08/01/fd90749a-b229-11e9-8949-5f36ff92706e_story.html
======
tptacek
Just to clarify: this article is against _gold standards_ , not against _peer
review_. It concludes by observing that peer review is a necessity. The
argument it raises is that peer review doesn't mean what lay audiences think
it means.

I think they're clearly correct, and that even on HN, in a community that
(roughly) prides itself on being scientifically literate, there are broad
misunderstandings of what peer review means (during the bogus "Sokal Squared"
hoax, for instance, many commenters implied that peer review prior to
publication was meant to encompass replication). Also, while I'm not a
"scientist", I've gotten to do some peer review work for ACM and Usenix, and
even in the little bit of review I did, I seen some shit. There is much less
formality and oversight to review than you might expect.

~~~
ZhuanXia
Is there any proof peer review has increased scientific productivity? It is
relatively new, and I don't see any reason prima facie why it would work
better than an editorial teaming willing to delegate when necessary.
Anecdotally, I see lots of academics complain about very important results
being unable to make it through peer review. Ralph Merkel's early
groundbreaking cryptography work was suppressed for years by unthinking peer
review. From Merklel:

>My first paper (and, in fact, the first paper) on public key cryptography was
submitted in 1974 and initially rejected by an unknown "cryptography expert"
because it was "...not in the main stream of present cryptography
thinking...."

~~~
michaelt

      Is there any proof peer review has
      increased scientific productivity?
      It is relatively new
    

According to [1] "The council minutes of 1 March 1665 made provisions for the
tract to be revised by members of the council of the Royal Society, providing
the framework for peer review to eventually develop, becoming fully systematic
as a process by the 1830s."

Peer review is a relatively new procedure in the same way that California is a
relatively new state :-p

[1]
[https://en.wikipedia.org/wiki/Philosophical_Transactions_of_...](https://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#Origins)

~~~
nextos
However, some famous papers like Watson & Crick's DNA model were published
without peer review.

~~~
asdff
You can publish today in PNAS without review if you are a member of the
national academy of sciences.

~~~
jhbadger
Not exactly any more. It's true that it used to be that a member could just
contribute their own manuscripts which would be published as is without
review. But after some unfortunate cases of questionable stuff getting
published, even the contributed track has to be peer reviewed these days.
However members still have leverage in that they can choose the reviewers,
which obviously could be abused.

------
elihu
Test results are very easy to fake. The "gold standard" is whether outside
groups can run the same tests and reproduce the same results. Peer review
doesn't mean much more than "this paper seems interesting and is well-written
and properly formatted."

~~~
mxcrossb
I disagree. The gold standard is if outside groups subject your findings to
_new_ experiments and reproduce the same results.

~~~
Fomite
I agree with you.

"I can run your code/experiment/etc. in your population and get your results"
isn't the gold standard either. "We approached it from a different way, and
with a different population, and we get a consistent answer" is both harder
and more compelling.

------
tomlue
I'm a recent PhD graduate that focused heavily on publication and collected
~20 publications. I think the situation with peer review is worse than
discussed here.

Peer review:

1\. dramatically reduces the pace of progress

2\. exacerbates publication bias

3\. creates a false sense of accomplishment

4\. creates a false meritocracy

5\. creates many perverse incentives

(3) and (4) unfortunately wrap into grant financing as well. The merit of your
research isn't measured on its impact on the human condition, but on it's
'impact' factor (a measure derived from your publications).

The gatekeeper problem here is pernicious as well. If you become a highly
cited author your ability to get/maintain financing improves. You also become
a gatekeeper as a peer reviewer. Which means that you are now strongly
incentivized to accept papers from people who cite your work or align
themselves with you, and reject everything else.

What is absolutely amazing here, is that the peer review process is opaque. It
is my belief that if you knew who reviewed which papers you would quickly
discover that mild to severe abuse of peer review is the norm, not action by a
handful of bad actors. This is because the entire academic reward system is
wrapped into the process. Getting your name on a big paper can have lifelong
ramifications on your ability to get grants, start companies, do consulting
work, etc.

Peer reviewers should probably get paid for the work. If they don't get paid
then their incentive to do review must come from somewhere else, vague notions
of improving the field don't cut it. Peer review should obviously be
transparent. Some people might be uncomfortable signing their name to a paper
rejection, but its time to get over that. A small payment might help reviewers
overcome this discomfort. It is bizarre that peer reviewers don't get paid.
Peer review is valuable work if done right, and without payment all the reward
of being a peer reviewer comes for the wrong reasons.

Finally, I agree with some other comments. Publication should not be
contingent on peer review, it should come first. This would increase the pace
of progress, reduce publication bias, reduce the false meritocracy, reduce the
ability for bad actors to censor research, and more. The cost would be a
larger number of publications, but perhaps this would help people realize that
many of the publications coming out right now are of little value.

Transparency is badly needed in many facets of academic research. My company
made a site that helps bring transparency into literature review (sysrev.com).

~~~
cassowary37
Those are all very real problems (and as you point out, they impact grant
review as well). On the other hand, fully transparent peer review doesn't
necessary address most of them. Reviewers who have to sign their reviews may
be reluctant to anger colleagues (or try to curry favor with them).

As you presumably know a many journals have experimented with open peer
review, but editors still need to police the reviews to look for bias. It
solves some problems but creates others.

~~~
tomlue
transparent review isn't a perfect solution. But the problem you suggest of
reluctance to anger colleagues or currying favor are worse under the current
system than they would be under a transparent one.

Just because other journals failed in the past doesn't mean we shouldn't try
again in the future! Maybe there were some mistakes we could learn from? Also
the internet is becoming a more familiar tool used more often by more people
every day, maybe it just wasn't the right time previously.

~~~
Fomite
Double-blind peer review fixes "reluctance to anger colleagues or currying
favor" and is widely used by journals in my field.

~~~
Nemo_bis
Maybe.

> In the late 2000s, widespread debate and controversy ensued after Budden and
> colleagues (2008a) found that a switch to double-blind review in the journal
> Behavioural Ecol- ogy led to a small but notable 7.9% increase in the
> 2proportion of articles with female first authors

[https://www.eswnonline.org/wp-
content/uploads/gravity_forms/...](https://www.eswnonline.org/wp-
content/uploads/gravity_forms/23-b28d66b6400f67d9648a049f8faf44e0/2015/05/Darling-2015-Doule-
blind-peer-review.pdf)

Others agree:
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5629234/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5629234/)

However:

> Findings from studies of journals that have actually adopted the practice
> are non-conclusive. For example, Budden and colleagues show that the
> introduction of double-blind review by Behavioral Ecology was followed by an
> increase in papers with female first authors (Budden et al., 2008). However,
> a second paper reports that female first authorship also increased in
> comparable journals that do not use double blind review (Webb et al., 2008).
> A study by Madden and colleagues shows that the adoption of double-blind
> reviewing in the SIGMOD conference led to no measureable change in the
> proportion of accepted papers coming from “prolific” and less well-
> established authors (Madden and DeWitt, 2006), but a later study contests
> this conclusion (Tung, 2006).

[https://www.frontiersin.org/articles/10.3389/fnins.2015.0016...](https://www.frontiersin.org/articles/10.3389/fnins.2015.00169/full)

------
pacbard
What would be the alternative to peer review? White papers on “pop” topics?
Opinion pieces by think tanks?

The ideal of peer review is pretty much the gold standard of scientific work.
By that, I mean the idea that your work should undergo rigorous scrutiny by
your peers before it gets accepted for publications. As any other human
system, though, the process is still subject to politics and biases.

My hope is that a truly open review system might one day democratize science
and ameliorate the issues with the current review system. For example, it
would be cool to put all the revisions, reviews and authors’ comments in the
open as an online appendix. That will show how the paper changes as it moves
trough the process, the issues that the reviewers brought up, and how the
authors responded to them. It will also help show if there is systemic bias
against certain researchers or topics.

~~~
nine_k
I think that _publication_ should not be gated by peer review. Release early,
release often, publish experimental setup and what you are looking for prior
to the actual experiment, etc.

We have arxiv now; the whole process could open up even more, slightly closer
to research in the open, like open-source development in the open. This is to
make peer review easier.

Then what is currently "publication" becomes what it should be, in the sense
of adding value: _curation,_ filtering the most interesting, least dubious
results.

~~~
chrisseaton
arxiv is actually _less_ open than most traditional venues.

Anyone can submit to most conferences and journals. Anyone at all! No PhD or
reputation needed and it’s blind. All that matters is the quality of the
research.

To submit to arxiv you need to be approved by someone as a legitimate
researcher, or beg for reviews from people you’ve never met without any
anonymity.

It’s somewhat a step backwards in some ways.

~~~
human_scientist
> To submit to arxiv you need to be approved by someone as a legitimate
> researcher, or beg for reviews from people you’ve never met without any
> anonymity.

I did not have to do something like this.

~~~
chrisseaton
Well that’s good for you isn’t it! What about everyone else? Not everyone is
automatically approved because of things like their email address. That’s my
point - they focus on reputation and credentials rather than the blind value
of the work. It’s less accessible.

~~~
317070
The confusion comes from the 'journal' aka topic on arXiv, which can have
different settings for when an author can submit. Some are more stringent than
others. It is not an arXiv wide issue.

------
fastaguy88
The idea that in scientific publication, more is better, makes little sense to
me as a practicing scientist. The scientific literature is already filled with
papers with conclusions that are not well supported by the data, and that is
after 50-80% of submissions are rejected.

To argue that publication should have fewer restrictions, one needs to show
that the modest number of papers that are improperly rejected by every journal
(not just the first submission) equals or outnumbers the overwhelming number
of submissions that should not be published because they reflect a
misunderstanding of their field or a misinterpretation of the data. For many
(most) scientists, the problem is not that important results are unpublished,
the problem is that it is almost impossible to keep up with the current gated
literature, particularly when that literature is full of mistakes. While it is
certainly true that some reviewers are biased and some ill-informed, most of
the time papers are rejected because the authors did not communicate well (for
papers that should be published) or because the authors did not understand
that their data did not support their conclusion (papers that should be
rejected).

The scientific literature needs a better signal to noise ratio, not more
noise.

~~~
Nemo_bis
The rejection of a submission does cancel a work, it merely shifts elsewhere.
The question is whether peer review gives the right incentives; many suspect
that publication venues with an ostensibly stricter peer review actually
encourage the production of more noise, not less. See
[https://www.frontiersin.org/articles/10.3389/fnhum.2013.0029...](https://www.frontiersin.org/articles/10.3389/fnhum.2013.00291/full)

------
mathgenius
> A 2015 study of 1,000 medical journal submissions found that of the papers
> that were eventually published, the 14 that became the most frequently cited
> were initially rejected. Groundbreaking studies by Sir Frank MacFarlane
> Burnet, Rosalind Yalow, Baruch Blumberg and others were rejected by peer
> reviewers, yet later led to Nobel Prizes.

Yup, if you are actually doing good work it is likely to be so different to
what is going on that people have a hard time evaluating it.

~~~
dagw
The question is why where they rejected? My wife is a researcher and does a
fair amount of peer review and sometimes she has to reject papers, not because
their necessarily wrong, but because they're so badly written its hard to tell
if they're right or wrong. Other times the research and paper is fine, but the
conclusions are overstated, for example claiming that A is better than B, when
all they've actually proven is that A is not worse than B. Other times it's
great paper in every way, but covering a topic that really isn't relevant for
that particular journal.

Basically there are lots of reasons to get rejected that have nothing to do
with the quality or validity of the research.

~~~
Fomite
I recently rejected a paper as "This is fine, and really interesting, but no
one who reads this journal is going to get it".

~~~
theaeolist
One of the problems with peer review is that behind the veil of anonymity
people can get away with such presumptuous nonsense. You are part of the
problem.

~~~
Fomite
An extremely technical paper on centrality metrics in a clinical journal is
_not the right audience_.

I suggested there were ways to approach writing it for that audience, but
they'd need to extensively rework the paper. The reason I know that? I've done
the same thing, for the same journal.

------
dan-robertson
A curious thing that’s happened in some parts of mathematics is that formal
peer review, needing coordination of journals, editors, and reviewers, can
happen after many people have already read the paper (effectively informally
peer reviewing it but with fewer comments), and even after people have cited
the paper.

This peer review then seems to not serve much purpose.

An example is the recent first finite prime gap paper where the result was
quickly strengthened several times by other authors long before the paper made
it into any journal.

Another example here:
[https://mobile.twitter.com/adrianprw/status/1156534906618597...](https://mobile.twitter.com/adrianprw/status/1156534906618597377)

------
alexmlamb
"Studies have shown that journal editors prefer reviewers of the same gender,
that women are underrepresented in the peer review process, and that reviewers
tend to be influenced by demographic factors like the author’s gender or
institutional affiliation. "

Most venues are double blind, so reviewers wouldn't know about the latter
things. Having fewer women in the reviewer pool is unfortunate, but it's hard
to see how it would effect the mean review quality in a direct way assuming
that men and women are equally good at reviewing (maybe in an indirect way it
could harm reviews by increasing the load per-reviewer).

~~~
bmc7505
Not sure if this is the case, but I'd like to play devils advocate. While most
venues are double blind, it is still possible there is a subconscious gender
bias. Many studies have shown that gender can be readily detected from
anonymized authors based style alone.

[https://www.aclweb.org/anthology/W11-0310](https://www.aclweb.org/anthology/W11-0310)

[http://cs224d.stanford.edu/reports/BartleAric.pdf](http://cs224d.stanford.edu/reports/BartleAric.pdf)

[https://www.sciencedirect.com/science/article/pii/S187705091...](https://www.sciencedirect.com/science/article/pii/S1877050916326849)

------
cassowary37
For everyone complaining about peer review, another alternative to open review
is paid reviewers who focus on a particular topic. For example, I'm an editor
at a journal where statistical reviewers are paid and evaluated, while the
remaining (content) reviewers are not. This ensures that we have a rigorous
review of the statistical methods, at least. I do think payment increases the
reviewers' responsibility and conscientiousness, and the overall quality of
the subsequent articles. (nb This probably has been tested somewhere.)

BUT - paid reviews are not feasible for many journals, especially open
access/low-cost journals where margins may be thin. (Conversely, Elsevier
should be able to pay their reviewers in gold bullion, rather than taking
advantage of the scientific community's altruism, but that's another
conversation).

------
typon
I think the peer review system is ripe for, as SV people call it,
"disruption".

Even something like an app that you submit papers to, which then anonymizes
them, standardizes formatting and presents them to randomly selected experts
for review in a certain time period would go a long way compared to the
current process. In my lab at a prestigious Canadian university, the papers'
authors names were visible as well as the reviewers names because any back and
forth post review would happen through email. I think journals and
universities need to buy into a system that standardizes this process so at
least these basic requirements of impartiality can be met.

~~~
tastroder
While I'd apreciate a little fresh air in the process as well...

> an app that you submit papers to, which then anonymizes them, standardizes
> formatting and presents them to randomly selected experts for review in a
> certain time period

Except for that it's not an app (that would be kind of pointless, why would I
want that on a phone instead of a webapp), that's pretty much the standard in
my field. Anonymization is done through the authors (if the conference/journal
wants it to be) and ensured by the conference submission system. Formatting is
standardized anyway.

The problem is the "randomly selected experts" part, for many (most?)
subfields suitable reviewers are a scarce resource, to the point that in quite
a few double-blind processes I've seen one or both parties could recognize
each other's work or review.

I've always found openreview [1] a nice way to at least make the process
public, especially when anonymity is an illusion anyway.

[https://openreview.net/](https://openreview.net/)

~~~
typon
By app I meant software not a phone app.

Secondly, I think the problem of having few experts in the field is a
legitimate problem but I don't think the status quo is a good solution either.
I think it erodes trust in the scientific community which is very hard to
build back up.

~~~
tastroder
I think we need an overall shift in incentives in academia before that will
change. Popular conferences see an all time high in submissions, though I do
not have data on that I'm sure Journals face similar problems these days. I
feel like that's not an aspect that will get better in the short term.

On the app/phone app thing: Ah, fair enough, sorry about that.

------
wtvanhest
I'm going to propose something that is unconventional today, but I believe
will be standard practice at somepoint.

Overtime we have shown that many mistakes are made in labs and we know that
many experiments are not replicable.

I think its time that we ask researchers using public funds to film their
entire experiment from beginning to end. Every pipette, every prep etc.

Peer review should also have a video audit. Then a replication in another lab.
The cost of ensuring results are what they say they are would be offset by
less wasted research dollars on false rabbit holes.

~~~
Nemo_bis
Are you trying to re-invent
[https://en.wikipedia.org/wiki/Reproducibility_Project](https://en.wikipedia.org/wiki/Reproducibility_Project)
?

~~~
wtvanhest
No. Reproducibility requires double the investment and may also have the same
or different mistakes. A video log would help researchers find errors and
confirm results.

Video is simply a better solution.

------
chr1
The whole scientific publishing field is still stuck in previous century. We
desperately need a github for science, where any reader would be a reviwer,
who would be able to rate the article and comment on it, and then journals
would use these ratings combined with credibility rankings of reviewers to
create indexes of interesting new article-repositories.

~~~
HNisCurated
I have people citing research with useless AMA standard. If you want both
parties to agree, they need to see the Data.

Most studies become uninteresting after you read their terrible data.

~~~
chr1
Absolutely agree, articles without data and without the sourcecode, are not
science, and that's another reason why we need github for science.

------
neonate
[http://archive.is/UCSof](http://archive.is/UCSof)

------
known
Is
[https://en.wikipedia.org/wiki/Wisdom_of_the_crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)
better?

------
jedberg
Somewhat tangential, but are there any journals that will only accept a paper
if someone unrelated to the author can replicate the study? Or a journal of
only replicated studies?

~~~
Nemo_bis
Sure. My favourite is IPOL Journal · Image Processing On Line
[https://www.ipol.im/](https://www.ipol.im/), which is doing it since 2011:
«IPOL is a research journal of image processing and image analysis which
emphasizes the role of mathematics as a source for algorithm design and the
reproducibility of the research. Each article contains a text on an algorithm
and its source code, with an online demonstration facility and an archive of
experiments. Text and source code are peer-reviewed and the demonstration is
controlled. IPOL is an Open Science and Reproducible Research journal».

More recently, some megajournals have joined the trend and are trying to
expand it, particularly eLife:
[https://elifesciences.org/labs/ad58f08d/introducing-
elife-s-...](https://elifesciences.org/labs/ad58f08d/introducing-elife-s-
first-computationally-reproducible-article) (also covered in
[https://www.nature.com/articles/d41586-019-00724-7](https://www.nature.com/articles/d41586-019-00724-7)
).

For some context see: *
[https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)
*
[https://www.frontiersin.org/articles/10.3389/fninf.2017.0006...](https://www.frontiersin.org/articles/10.3389/fninf.2017.00069/full)

------
scythe
We don’t.

Peer review is a very helpful component of a system that slowly filters and
refines academic work. It’s like the first air intake stage of a jet turbine.

------
p1necone
If peer review in Academia is anything like code review in corporate software
dev it's probably next to worthless.

~~~
jdnenej
Peer review never seems to catch actual bugs but does give you helpful tips
like "we already have a function that does this" or "the language has a
feature for this"

~~~
p1necone
Oh code review done properly is really great - I'm just jaded by repeatedly
seeing code go up and getting that green tick with no comments and knowing
that the reviewer barely even attempted to read it.

