
Nature will publish peer review reports as a trial - aaavl2821
https://www.nature.com/articles/d41586-020-00309-9
======
scott_s
I think this is good, and may also _improve_ reviews. I doubt that Nature has
an issue with substandard reviews, but I have definitely seen it in some of
the computer science conferences I submit to and have served on [1]. If
reviewers know that their review may be published, I think they may turn in
more substantial reviews.

[1] In computer science, conference publications are peer-reviewed and highly
competitive. We also have journals, but most novel work appears in
conferences. The more theory oriented parts of computer science will tend to
publish in journals more, but they still publish in conferences, too.

~~~
biomcgary
Published reviews will just make reviews political in new ways. As a graduate
student, I helped my advisor review a paper submitted to Nature that was
publishing a large biological data set. The internal metrics that the authors
provided made it clear that the data was 95% noise. The review was direct and
to the point. The paper was rightfully rejected, but ended up in another high
profile journal after the authors removed the damning evidence. The
corresponding author was one of the most famous and powerful people in a very
large field. My advisor was still a relatively junior professor at the time,
even if well known. I don't think publication of the review would have changed
things for him, but for many others I have known, it might have.

~~~
beerandt
I'd say this is actually a stronger argument _for_ publishing reviews, as a
persuasive argument might prevent venue shopping, potentially in multiple
ways, depending on the when.

~~~
abdullahkhalids
Ummm. The reviews are only published for the accepted papers - and everything
is optional now. So rejected papers can still submit to another journal.

------
tcpekin
As mentioned in the article, Nat Comm has been doing this for a few years now.
I have found it immensely interesting and helpful for my own research to have
access to the peer review files (usually in the Supplementary Information
[1]). It is a great way to show younger students the process and formatting
behind peer review and also have examples of what constitutes a positive
review process. As an author, it does feel like a positive as well to have the
peer review file out, so that readers who are interested can see how the work
developed, and what points or weaknesses the reviewers focused on.

[1] randomly chosen from homepage -
[https://www.nature.com/articles/s41467-020-14530-7#Sec11](https://www.nature.com/articles/s41467-020-14530-7#Sec11)

------
mxcrossb
We all know that publishers add little value to science, and nature here looks
to be striving to add even less than usual. If the peer reviews are not of
good enough quality, your editors should be handling that! Don’t rely on the
readers to review the peer review. Dumping the peer reviews online is just a
way to avoid the responsibility of quality control.

~~~
chaos_emergent
Conversely, it's also a way for the general public to see criticisms and
laudations of research that might otherwise be accepted with blind faith.
While Nature has its pick of comparatively incontrovertible research, sending
a signal to the rest of the industry that peer reviews ought to be published
seems like a useful step toward better, more replicable, more honest science.

~~~
Vinnl
> While Nature has its pick of comparatively incontrovertible research

Nature doesn't actually attempt to select for incontrovertible research, but
rather the most groundbreaking/noteworthy research. Which is also why their
retraction rate (and those of other "top" journals) is higher than average.

(Disclosure: I'm part of an initiative that explicitly distinguishes between
exciting and robust research for this reason.)

------
yurlungur
I wonder what is the impact of this change.

From my limited experience peer reviews are treated as fairly private
communications between the reviewers, editor and authors. We would often use
it as an important clue on what to change even if we are going to another
journal for publication. This makes the content of the reviews rather
sensitive, especially in the cases where there is competition for publication
on the same subject.

There are also the cases where Reviewer X may ask authors to do a very
specific analysis/improvement that may somewhat benefit their own future
research. It's all par for the course.

Another aspect of this that sometimes experienced professors have a fairly
good sense on who's giving the review, even though they may be anonymous.

Since this policy gives authors the choice of publishing reviews, I hope it
does not degrade the quality of reviews and make people uncomfortable for
writing "rude" feedback.

~~~
prepend
I don’t publish or review too much, but I do a little. One of the recently
frustrating events to me is that I got comments back from a reviewer that made
an incorrect factual statement along the lines of “it’s against government
policy to share x type of data.” And then discounted large parts of the
conclusion as impossible or inappropriate. That was the only substantive
comment from the reviewer.

We responded back to the editors refuting it and showing the relevant policies
and regulations showing how data sharing for x was allowed.

The next message back was from editors declining. So further feedback.

I would have really liked to learn more about why the reviewer thought what
they thought and why what seemed like clear evidence to my co-authors and I
wasn’t to the peer reviewer and editors.

I’ll never know and I don’t spend many cycles contemplating it, but this new
trial in Nature would help me understand the criticism better.

------
cosmic_ape
Some comments seem to miss the point that reviews will be published without
naming the reviewers (unless they want to), and then only for accepted papers.
So for the reviewer this should not change things much.

Also, NeurIPS conference publishes reviews for a long time now. But I haven't
heard of anyone talking about a review of a paper. And if you do look at these
reviews, really many of them are of really dismal quality. Maybe things will
be different for Nature, idk.

~~~
captain_price7
Do you know any other ML conference/journal, doesn't have to be NeurIPS
quality,that also publishes reviews? I know there is ICLR, but anything else?
I really like reading them.

~~~
cosmic_ape
Not that I know of. But its seems a few other conferences use OpenReview,
besides ICLR. UAI for example.

------
bluenose69
Start by publishing the name of the editor alongside each paper. Some editors
barely look at papers or even the reviews. They basically just look at whether
the reviewers ticked the 'accept' or 'reject' box, and base the decision on
that (after possibly getting another review to break a tie). This can go
undetected for quite a while, unless the journal has a chief editor who keeps
on top of things.

------
ISL
Oh, man. I've been involved in peer-review discussions where the review
correspondence is far more voluminous than the article itself. I wonder how
the journals will handle the page-count.

~~~
chr1
They should publish it only in electronic version.

~~~
petschge
That does not solve the problem that more pages take longer to read. Given the
pace of science these days keeping up with papers just in your subfield is an
actual problem.

~~~
mr__y
Do you actually read whole papers? It's an honest question. I usually scan the
introduction, quickly skim through charts/tables and jump directly to
conclusions. If I find anything remotely interesting during that process, then
and only then I would even consider reading the whole paper.

~~~
petschge
I probably read about three full papers per week. The rest I skim similar to
you.

------
esfandia
I like the idea in principle, but it raises the bar for the quality of the
review that is going to be published, and might thus discourage people from
accepting to review papers. It is hard already as it is to find peer
reviewers!

We're getting to the point where there really has to be a financial incentive
to do a peer review (Springer gives you a free book, which is a start). If
that were the case, I wouldn't mind doing it occasionally when I retire.

~~~
yqx
> We're getting to the point where there really has to be a financial
> incentive to do a peer review

IMHO, a free book is an insulting and scientists shouldn't accept to do this
work for _commercial_ publishers for free to begin with.

I know many people feel it's their responsibility to provide "services to the
community" which is great, but it's ridiculous that commercial publishers get
to profit from this when they proceed to claim ownership of academic work and
hide it from the public behind a paywall.

If they want to make money off research, fine. But reviewing costs time and
money. Academic publishing may well turn out to be not so profitable anymore
if reviewers have to be paid. Maybe then we can finally move to university-
hosted open-access publishing.

------
kissickas
> Research communities are unanimous in acknowledging the value of peer review

> 82% agreed that standard peer review ensures high-quality work gets
> published

How common is the view that peer review only serves as an institutional
gatekeeping mechanism and harms innovation? (as I've heard Eric Weinstein
charge on his podcast, _The Portal_ )

~~~
coughupalung
EW has a PhD but I can't seem to find any academic publications under his
name. That doesn't outright mean he's wrong but it does mean that he actually
has limited experience with the process. I have several and I've reviewed
quite a few papers. I would say that peer review is a pretty good filter for
garbage. Typically reviewers want to see shiny new ideas so if anything it
over emphasizes innovation and discourages incremental improvements. As for
gate keeping, I don't think so. Again, reviewers want to be WOW'd with new
ideas.

In general, I prefer papers from peer reviewed venues over the arXiv. Good
work does get put on the arXiv though so I can't ignore it completely.

~~~
bumby
> _Typically reviewers want to see shiny new ideas so if anything it over
> emphasizes innovation and discourages incremental improvements._

I know publications fetishize novelty, but wasn’t there research showing a
high percentage of publication is either non-replicable or derivative and
subsequently of little value once you remove auto-citations?

------
buboard
Elife did this experiment long ago (one of their many innovations)

[https://elifesciences.org/inside-elife/e9091cea/peer-
review-...](https://elifesciences.org/inside-elife/e9091cea/peer-review-new-
initiatives-to-enhance-the-value-of-elife-s-process)

------
kaliali
Why not have a site where people can openly comment and discuss individual
research papers? This would make it more accessible to the public and increase
the transparency in the research. You can have a mod that keeps the
conversation on topic and in any case it's better than nothing.

~~~
joshvm
This is somewhat the point of openreview, except the reviewers are still
anonymous.

[https://openreview.net/](https://openreview.net/)

------
isaacyes
The British Medical Journal (BMJ) have been doing this for a while -
publishing both the peer-review comments and author responses. I have
published a paper in the BMJ and thought the process worked well and tempered
the responses on both sides.

------
Fomite
It saddens me somewhat that in the push between open peer review and double-
blind peer review, open peer review seems to be winning.

I've had a couple bad experiences with it, and _vastly_ prefer double-blind.

~~~
timkam
In many communities, the problem with double-blind peer review is that it's
pretty clear who the authors are, based on the exact line of research, writing
style, et cetera. And in many cases, you can also guess who the reviewer is
with some confidence. This means that double-blind is a nice feature for the
"big guys" who dominate their communities to show how objectively strong their
research is, although in fact oftentimes their friends review their work and
know exactly who the authors are. Open peer reviews would at least make sure
that there is some degree of accountability.

~~~
Fomite
That's potentially true. However, I will say that as a reviewer for several
double-blind journals, I've often thought "I totally know who is writing
this..."

I've been wrong every time.

------
xvilka
All research indeed will benefit from such an approach. Think of transitioning
from the lack of version control systems to their appearance. But it is not
enough for better scientific publishing. There was quite an interesting
discussion[1] about creating something in between Overleaf[2], ArXiv[3], Git,
and Wikipedia, moreover with the ability to do a peer-to-peer review,
discussion, and social networking. See also the last[4] article in that
series. There are a few implementations, albeit not covering all features,
like Authorea[5] and MIT's PubPub[6] (it is the open source[7]). See also
GitXiv[8]. See also the Publishing Reform[9] project. Moreover, there is quite
an interesting initiative from DARPA, to create the scientific social network
of a kind - Polyplexus[10].

[1] [http://blog.jessriedel.com/2015/04/16/beyond-papers-
gitwikxi...](http://blog.jessriedel.com/2015/04/16/beyond-papers-gitwikxiv/)

[2] [https://www.overleaf.com/](https://www.overleaf.com/)

[3] [https://arxiv.org/](https://arxiv.org/)

[4] [http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-
up-a-...](http://blog.jessriedel.com/2015/05/20/gitwikxiv-follow-up-a-path-to-
forkable-papers/)

[5] [https://authorea.com/](https://authorea.com/)

[6] [https://www.pubpub.org/](https://www.pubpub.org/)

[7] [https://github.com/pubpub](https://github.com/pubpub)

[8] [https://medium.com/@samim/gitxiv-collaborative-open-
computer...](https://medium.com/@samim/gitxiv-collaborative-open-computer-
science-e5fea734cd45)

[9] [https://gitlab.com/publishing-
reform/discussion](https://gitlab.com/publishing-reform/discussion)

[10] [https://polyplexus.com/](https://polyplexus.com/)

------
Vinnl
Note that eLife will also be publishing peer review reports, even independent
of "accepting" or "rejecting" a paper (and might even abandon that concept
altogether):
[https://twitter.com/mbeisen/status/1155286615721254912](https://twitter.com/mbeisen/status/1155286615721254912)

(Tweets by eLife's Editor-in-Chief.)

------
nestlequ1k
They've also announced two days ago that they will have a new journal
dedicated to aging. This is big news for those of us who want longer,
healthier lives.

------
mmhsieh
there should be a public and a private portion

------
adamnemecek
The peer review process has reached a point where it's more harmful than
helpful.

What I want has following properties:

* Version control, I can contribute to other people's papers

* I can link to a sentence in another paper at some point in time so that when I read another paper, I can go to the location immediately

* Open, like arxiv (anyone can publish anything), however arxiv discourages uploading personal or class projects, which is a terrible idea, the best part about GitHub is finding someone's abandoned project that does something

* Full-text search

I think that most academic publishing startups approach the problem by
indexing already published papers. I think that starting clean slate is
achievable.

~~~
ivalm
This may work for some, but not many types of work.

For example

> * Version control [...]

If I am working on a non-trivial system made of some novel-ish material doing
some hard-ish measurements (I just described most of experimental
physics/chemistry/material science) then there is approximately zero other
people in the world who can reproduce without considerable effort.

This reproduction effort is meaningful/justified once the original group finds
some cool effect and ensures it is not spurious, but that's relatively late in
the paper development (and at a point where it would make more sense to write
a new paper for follow up rather than contribute to the current work).

~~~
adamnemecek
Maybe they can fix spelling, maybe they can add graphics, or fix wording.

------
glofish
The current scientific publishing method is pretty much dead. It is absurd,
biased, produces terrible results. Promotes inept research, unimaginable
waste.

Yet dead as it might be most decision-makers and other leeches of science will
pretend that it is just fine and alive. They will continue to do so for many
years. Decades perhaps.

It is like watching Weekend at Bernie's.

This attempt of Nature is little more than holding up Bernie, waving his hand.
Look, it is moving! It is not dead.

~~~
coughupalung
How about you do something constructive instead and propose a better
alternative. You're just "poisoning the well" with empty accusations and
ignoring how successful the scientific method has been.

~~~
chr1
The parent comment is not an empty accusation, but a truth known to everyone
in academia. The publishers extort huge profit margins from libraries and
universities without providing any useful service. See
[http://thecostofknowledge.com](http://thecostofknowledge.com) and the
articles linked from it for more information.

~~~
coughupalung
That's not what the OP said though. I am in academia and most of my colleagues
would agree with you about publishers. That's why most of us publish pre-
prints on our websites, so that everyone can access our work. We can debate
the business model but the OP is saying it produces inept research without
really supporting that statement.

~~~
glofish
Would this serve as a supporting statement?

Why Most Published Research Findings Are False

[https://journals.plos.org/plosmedicine/article/file?type=pri...](https://journals.plos.org/plosmedicine/article/file?type=printable&id=10.1371/journal.pmed.0020124)

It was true in 2005 and the situation only got worse since.

