
Peer review: the end of an error? - montrose
https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/
======
throwawayyx96
Author of the article is the same guy who, some years ago, organized a boycott
of Elsevier journals due to their extortionate charges.

[https://en.wikipedia.org/wiki/The_Cost_of_Knowledge](https://en.wikipedia.org/wiki/The_Cost_of_Knowledge)

He has practiced what he preaches with respect to publishing in open access
journals.

~~~
VHRanger
Timothy Gowers has the moral courage to do what pretty much everyone else is
waiting for other people to initiate

~~~
gowld
Being recognized as the top living mind in an entire field of science
backstops a lot of courage.

------
improbable22
A point not mentioned in the article is that this formal peer review system is
itself quite young. I think that sending every paper out to several reviewers
only became the default once we had xerox machines -- 1960s? We did a lot of
serious science before that.

A bit earlier, the famous anecdote is about Einstein's offence that the Phys
Rev editor would dare to send his paper for review at all -- apparently this
would not have happened at a German journal of that era:
[http://www.preposterousuniverse.com/blog/2005/09/16/einstein...](http://www.preposterousuniverse.com/blog/2005/09/16/einstein-
vs-physical-review/)

~~~
noobermin
If anything, that works as a great defense of peer review given Einstein was
wrong.

~~~
gophile
Einstein was right about some things and wrong about some things. Which work
of Einstein are you referring to when you imply that a peer review could have
detected the error in his work?

~~~
noobermin
The story in the linked blogpost.

------
jseliger
One hopes the present system will go away:
[http://sappingattention.blogspot.com/2017/09/peer-review-
is-...](http://sappingattention.blogspot.com/2017/09/peer-review-is-younger-
than-you-think.html)

The problems are in some ways even worse in the humanities:
[http://www.chronicle.com/article/The-New-
Intellectuals/23835...](http://www.chronicle.com/article/The-New-
Intellectuals/238354?key=dCUViR9a-EOigReN8KYScI_cuA0Bu5caKtkfgTzhUtPxUY_oOHNXK3YWfjxQpQr4VlF0LURGdjBDZFZTdUQwMUR3VkI5b2lWRW55Q1hicXUyc0pMWDF5eTFZUQ)
than the sciences. Or see [https://jakeseliger.com/2013/02/12/a-lot-of-
academic-researc...](https://jakeseliger.com/2013/02/12/a-lot-of-academic-
research-may-barely-count-as-research-by-many-definitions) .

------
woliveirajr
> Remarkably, we have arrived at a system where academics feel a moral
> obligation to perform the thankless task of reviewing the work of other
> academics, anonymously and unpaid. This undoubtedly makes the literature
> better than it would otherwise have been, and ensures at least one reader
> for each paper.

I'm afraid it doesn't ensure that all papers will be read. Some papers can
attract more attention, some papers can even receive too much comments with
relevant comments being drown in the sea of observations.

~~~
John_KZ
Exactly. It's the same fallacy with open source code. Just because it _could_
be reviewed, doesn't mean it has been. Formal or semi-formal groups that
professionally review papers are essential. How are they going to get paid? If
you want research to be publicly available (as it should be) you can either
put the reviewers on government dime (which isn't a great idea but it's better
than nothing) or you can follow the system that was pretty much followed until
now:

Research institutions that produce papers to be reviewed, pay up the
reviewers, and then we make the research publicly available, because everyone
is paid and happy. No need to double-pay the reviewers via per-paper costs
etc. The cost of reviewing a paper is fixed. Hosting costs are trivial
compared to that. The uni can take them on. In a sense sci-hub fixed the peer-
reviewed industry. Formalize it (make all journals open-access) and we're
done.

~~~
isolli
Pull-request reviews work the same as peer review: it ensures that at least
one person (the reviewer) reads the code or article.

------
danieltillett
The big problem with peer review is almost no reviewing takes place and it is
at best is either a way for your enemies to get revenge or a cheap
proofreading service.

The solution is to offer a largish reward to the most constructive peer review
for each manuscript chosen by the editor, or better by other reviewers. If I
(or more likely one of my starving grad students) knew that I would get $1000
for providing the most constructive peer review then I would put in the
required effort to make peer review a worthwhile exercise.

One thing that would help the peer review process is a data dump of the raw
data. Many times I have seen some figure or table in a manuscript and felt I
could not provide a constructive critic without seeing the underlying data.
Going back through the editor to get this data is just too much of a hassle
most of the time.

~~~
wyattpeak
But who decides what's constructive? It's all very well if you're choosing
amongst people who basically agree with you - what if they're telling you it's
going to take another year of experiments to get it up to standard?

Which reviewers, too, do you respond to? If it's a popular paper you probably
can't respond to all criticism, but picking which criticism to interact with
isn't going to lead to good science. There's a lot of value, even if it's not
perfect, in writers being forced to respond to one or two reviewers' comments.

~~~
danieltillett
I suggested in the post either the editor or the other reviewers makes the
decision of which was the most constructive review.

The suggestion of going back and doing more work exists under the current peer
review system. With my proposal you might even get suggestions of what work
needs to be done and why you should do it.

In all the papers I have published I have only had a few reviewers agree on
anything. The normal approach is you agree to all the reasonable demands even
if you don’t personally like them and argue over the important one. Most of
the time if the reviewers see you have moved some of the way towards their
arbitrary demands they will let you hold onto the important ones.

~~~
wyattpeak
You're right, I completely missed that. I quite like the idea of other
reviewers picking which is best, although it may take some fiddling to avoid
abuse.

I wasn't suggesting reviewers currently can't tell you to go back and do more
work, quite the opposite - that I think it's a valuable if often too arbitrary
requirement. But if the writer isn't picking which comments to respond to my
point is moot anyway.

------
kelnage
The initial examples the author uses really do reflect the magnitude of the
third problem. All of them were big breakthrough results in their fields - and
hence people cared enough to do proper reviews - especially in the latter two
cases. As an academic, I have participated in reviews of papers that I would
have never otherwise read, but could still contribute to the process. Without
something pushing all academics to participate in the system, then a more open
solution risks moving science even closer to “only interesting
problems/solutions/results get published”.

~~~
aldoushuxley001
"Only interesting problems/solutions/results get published"

I'm afraid we're already there. Journals are very much strongly incentivized
to publish only interesting or novel results.

~~~
improbable22
Isn't this the point of journals? They communicate new knowledge, which
changes the status of what is known.

There's a balance to be struck in how certain these things should be before
being published, for sure. The p-hacking plague is a symptom of setting this
certainty bar too low, or equivalently, the required level of novelty too
high.

Things which stay constant are the job of standards bodies, and textbook
writers... these are important too, but different.

~~~
squiggleblaz
I don't think so. If you work in a hard science like medicine or psychology,
where problems are still so hard we can't produce clear answers as to results
of a paper, we shouldn't really be trusting papers until they've been
reproduced by some unrelated team. But if these results don't contradict the
previous papers, they're not really communicating new knowledge.

Sometimes this isn't a big deal (medicine is as far as I know regulated enough
that reproduction is a part of the problem), but most science is characterised
by not enough money distributed without caring about scientific outcomes.

So we do have a problem at the moment. A team who does important but thankless
scientific work can't get published. Too much science is just first draft
status. And the only way to determine that is that people keep trying and
trying to build on it and eventually decades later the tower of cards
collapses when clever new PhD students start down new research avenues.

------
jhbadger
I think the mathematical sciences see peer review as less of a requirement
than the experimental -- in theory, anyone can verify a proof for themselves.
But in experimental science you really need someone familiar with the
techniques to make sure that they did all the needed controls and didn't do
any shortcuts and that's what peer review is useful for.

~~~
noobermin
Was going to comment to say the exact same thing. OP used a paper by medical
professionals to poo-poo formal peer review and then used math papers as
examples of successful arxiv papers scrutinized by other scientists. After
realizing that, I stopped reading the article then Googled his name, and--
surprise, surprise--he's a mathematician.

It will work for mathy and theoretical sciences like CS and theoretical
physics, but good luck getting experimentalists to review papers for free (or
at least without being asked by a journal). I'm not sure whether peer review
is necessary in those cases, but the culture around more applied sciences
definitely considers peer review more important, even though we routinely post
to arxiv anyway.

~~~
gowld
It sounds like you're saying that experimentalists rely on peer review as a
way to fraudulently cover up their shoddy experimental methods.

~~~
noobermin
Hmm, if you're lab A, why would you help to do the cover up of lab B's work,
especially if you might be competing with them?

Scientists at least in my field don't do each others' dirty work, we're much
more disconected from each other than that. Also, peer review is anonymous.

------
probably_wrong
There are two points that I wished were addressed more often.

1\. New researchers. The examples he mentions include researchers who were
already important in the field, and whose prestige guarantees review. But if
you're a new researcher, good luck getting noticed. Blind review doesn't have
this problem.

2\. The cost of journals. People say "journal" and think of Elsevier, but
there are several journals and conferences that publish all papers for free.
The ACL, the largest conference on computational linguistics, finances itself
with the entry fee to their conference, and you can read all of their papers
right now (disclaimer: I'm currently doing work for them). There is a middle
ground between Elsevier and Arxiv.

~~~
femto
One way to look at journals are that they are "ready made" reputation for
those without their own. A new author can leverage the reputation of the
journal to establish their own reputation.

On that basis, maybe the mark of a truly prestigious researcher is one who can
post directly to arXiv? Rather than the highest accolade being for a paper in
Nature, the highest accolade should be for a widely reviewed paper on arXiv?

------
kolpa
The academic peer review debate has strong parallels to the social-media news
debate -- As traditional newspapers/ TV channels are no longer gatekeepers of
publication, we have given up a level of vetting in exchange for freedom-from-
censorship. But that puts an onus on consumers -- it's not right for consumer
to accept everything they see with the same level of default trust they had
before. Consumers still must "consider the source", and society to create an
effective replacement for the editorial filters we are discarding.

see also: "web of trust"

------
dunk010
I am afraid this piece is missing the most massive reason that articles are
still published in journals - Impact Factor. Academics live and die not just
by the number of articles that they get published, but, crucially, _where_
they get published. The entire academic system of progression is basically
controlled by the journals. The reason that the splashy websites never succeed
is that they have targeted the wrong problem. If you want to fix the system
then you have to come up with a new way of rating academic's publications,
which would then be fed back into the career progression system in their
individual institutions. Good luck with that.

~~~
iandanforth
While this is true in many institutions there are now useful proxies,
specifically a Google Scholar profile or an individual h-index which can be
calculated without publication impact factor.

You are correct that it will take a long time to update documentation and
policies that reference impact scores, but informally, at least in Machine
Learning, people know who the contributors are with high actual impact. In
addition because a lot of impactful work is coming out of industry and there
is rapid turnover between academia and industry the culture as a whole is
evolving.

~~~
dunk010
Of course there's an untruth in what I said. _Ostensibly_ it's all about
Impact Factor, but in reality it's all about how much money you are bringing
in. Academia is in a sad state.

------
wbillingsley
In some fields, such as HCI (and research through design) I think there could
be value in moving to a system where instead you version papers and also
publish the criticisms that led to its refinement.

If you've explored a problem, designed a piece of technology, and taken it
back to that audience to gain further insights into the problem, what I can do
in reviewing it isn't the same kind of analysis as a maths proof. Even if I
you were to give me the tech or the code, I wouldn't have the same people to
test it with, and would not expect to get exactly the same results. Instead,
peer review tends to criticise the writing, missing elements in the argument,
pieces of literature that also need considering, aspects of the methodology,
and suggest more things you could do.

All of which is very valuable, and sometimes peer review is the best (only)
mentoring you'll get if you're in a small university. But it's not quite the
same as "we have to review this or a wrong claim might get the imprimatur of
being science". Design questions don't make the same claims of completeness or
context-independence. It's more of an argument based in the kind of muddy
evidence you have to put up with when tackling muddy problems.

Which makes me think that instead of peer review being a hurdle to
publication, perhaps we should just make the whole argument and refinement
public. Give the reviewers more kudos as instead of being invisible entities
whose thoughts are only received by the author and editor, their role in the
development of that idea becomes a published item too.

------
pizlonator
I've published in CS conferences, CS journals, and on a blog.

Publishing in CS conferences/journals means you get reviews from 3-6 "smart
people", and then maybe 50-100 people will read your paper, and most of those
people will never contact you.

OTOH if you publish a blog and advertise on hackernews and twitter, you get
instant feedback, including from some of the same experts that would have done
reviews for conferences/journals.

~~~
tincholio
The difference is that more often than not, you can't go to the same level of
detail and depth in a blog post as you would in a scientific article (and if
you do, not that many people will read it). Reading science is hard work, and
properly reviewing a "journal length" paper can take upwards of a day. I don't
think you can have it both ways. If you write a good vulgarization, or
technical blog post, and submit it here, you will likely get many eyeballs. If
you post serious science, you'll get much less feedback, if any.

~~~
pizlonator
My blog posts typically have more scientific content than a CS paper.

~~~
tincholio
I'd be happy to see some, please link

------
xamuel
Not to defend peer review as it is now, but one of the things I've come to
value from editorial gatekeepers is that they put the brakes on me publishing
things that in later years I might regret. I wrote some pretty mediocre papers
as a grad student and I'm grateful some of them were rejected, because I
wouldn't want them forever part of my legacy. As a matter of fact, I wrote
some "papers" when I was _a teenager_ , which, if I had been allowed to freely
publish at that time, might have become permanent albatrosses around my neck
:P

~~~
danieltillett
Didn’t your supervisor step in and say this is not up to scratch? When I was a
supervisor a good chunk of my time was spent going through my student’s and
postdoc’s manuscripts to ensure nothing was in them that might embarrass us
both.

~~~
xamuel
I was reckless and wrote papers unrelated to my dissertation and submitted
without going through my advisor.

Fortunately, the bad ones were rejected. The good ones ended up making me
well-recognized in some extremely tiny niches, and ultimately did way more for
my career than my PhD work did :D I'm really glad some of the bad ones didn't
see the light of day :P

~~~
tom_mellior
> I was reckless and wrote papers unrelated to my dissertation and submitted
> without going through my advisor.

This is the opposite of reckless, this is how you make a name for yourself,
and how you prove that you can develop an independent research agenda. As you
say, it's good for your career, and it's good for science too (if the work is
good, of course). Good for you!

------
lowkeyokay
I really think we should have something like Stack Overflow for academic
papers. The comments system works so well. Also, the so called impact factor
of journals, which in many cases are directly or indirectly linked with
academics and universities financial outcome, could be replaced by the points
system from SO.

Of course I'm not proposing using SO directly for academic research but
something inspired by it.

~~~
hvidgaard
I think a hybrid between SO and Discourse for papers would be more fitting.
There really needs to be an open but moderated discussion for the papers -
that'll serve as the peer review.

------
CharlesMerriam2
An interesting parallel can be found in the anonymous review of grant
proposals. See "Fund ideas, not pedigree, to find fresh insight",
[https://bit.ly/2Fy64cP](https://bit.ly/2Fy64cP). It ends with "peer review
may be the worst system, except for all the others". It can be found in
Nature, a peer reviewed journal.

------
jwilk
Archived copy, which can be read without JS enabled:

[https://archive.is/iPwXi](https://archive.is/iPwXi)

------
freech
Reddit but only academics can participate? Probably better then peer review,
but it won't have the aura.

------
arca_vorago
I've said it before and I'll say it again.

 _The focus should be on reproducing studies, not on peer-review._

Of course peer review can be of assistance in selecting studies worth putting
the effort and funding into reproducing, but peer-review itself is extremely
corrupted as a process, but since so many peoples careers depend on it , many
of them will not talk about those issues.

I used to hold the peer-review and journal process on a pedestal. Then when
working at a genetics lab I ended up having to read a lot of papers to get my
sysadmin work done, and started realizing just how bad so many papers are,
even though published in "reputable" journals.

For example, since publishing is seen as so vital to some people, I noticed
many of them would "consult" on another groups paper just to get their name on
the paper, even though they did almost no work, and then those same people
would turn around and say stuff like "I've been published X times!" as a sort
of reinforcement of authoritativeness (and as advertisement).

People are gaming the system because hardly anyone is actually checking them.
Beyond that, the actual peer-review process itself seems to be very lackluster
in rigor on it own.

I think there should be a certain amount of public research funding that
should be earmarked to go directly to independent reproducability labs.

On top of it all, all this research that the public is funding is often behind
a fucking paywall... and that's not just bullshit, it's theft and misuse of
public funds. It should be required that publicly funded research should be
publicly published, and therefore available for more eyes to see and review.

~~~
dragonwriter
> The focus should be on reproducing studies, not on peer-review

The two aren't separable, since a key part of peer review is assuring that a
published study is, in principle, reproducible. You can't focus on
reproduction withoit focussing on peer review.

~~~
squiggleblaz
Reproducing is a form of peer review (if you assume that a person who
successfully attempts to reproduce your work is ipso facto your peer).

But the fact that your peer (i.e. another academic who works in the same area)
has read your paper and commented on it in the process of publication (i.e.
customary "formal peer review") does not reflect on its reproducibility. The
peer, being a human, has attentional deficits and there's probably true facts
that increase the chances of the result occurring by chance which are not
considered worthy of publication.

Until a paper's result is reproduced, it can't be regarded as reproducible.

------
jwilk
(2017)

------
chrisseaton
This article says that peer review is unpaid - it isn't.

If you are an academic or work in industrial research, it's part of your paid
job to do peer review as part of the community. You are paid for it - it's
part of your job and it's normal to do it during work hours.

~~~
heckless
Although reviewership is a useful thing that gets put on CVs, I can't think of
a position I've ever seen where "review X papers" a year is part of the job
requirement.

A more salient point about why it is important to note that peer review is
unpaid is that academics, as authors, submit their work---for free---to a
journal, where reviewers review it for free, and then the journal turns around
and sells the article for $35 apiece.

~~~
chrisseaton
This is such an adversarial way to look at it.

I see it as a community. We write and share our papers as part of our jobs, we
review each others' papers as part of our jobs, we help to run the conference
as part of our jobs. Everyone contributes to run the system.

Nobody buys articles for $35 a piece. All major universities and companies
subscribe to the ACM or whatever your field's organisation is for you. And
guess who runs the ACM and other organisations? The community does! People
from the community serve as leadership and help run things.

Yes the ACM gets some money to run their website, conferences and outreach
programs. I don't think that's crazy.

~~~
heckless
I don't have a huge issue with the ACM or even so much with IEEE. But have you
ever submitted to or reviewed for Springer or Elsevier? It's an emasculating
and thankless experience, and Springer and Elsevier go get to make huge
profits off of everyone's free work. (haha, we just did it for exposure,
right???)

I don't mean to imply that peer review is this shitty thing that we "have" to
do. I review papers regularly and generally enjoy the process. But I think
it's important to point out that no, we are not being paid for it, and
actually typically the incentives in our jobs mean that peer review is a
"thing we have to stuff in there somewhere", and certainly not a primary task.

~~~
kolpa
> emasculating

that's an unfortunate word.

