Ask HN: How do we fix peer review?

[panabee]

Peer review made sense in the paper age.

However, it is severely outdated in the Internet age when corrections and changes can be published and tracked instantly. Moreover, the noble goal of filtering out junk yields the undesirable consequence of conformism, gated by whoever happens to review a submission.

If the objective is to surface quality research, how do we update our systems to reflect technological advances?

The tradeoff of accelerated publishing is more misinformation, both intentional and unintentional. But the benefit of faster, more valuable knowledge reactions (chemical reactions but for knowledge) seems to merit deeper exploration.

[dane-pgp]

I was wondering recently whether there needs to be a system to accompany peer review, which allows scientists/reviewers to declare their biases on whatever scientific controversy exists in their field.

If a paper is published by a person or group that oppose the view of some influential experts, the writers shouldn't have to censor their paper in order to get it past review, but it should have to answer the criticisms of the opposing camp. Similarly, a paper shouldn't get a free pass just because it supports the reviewer's biases.

Taking things a step further, it should be possible to anonymously stake reputation on specific sides of these controversies, so that wise contrarians can be vindicated with time (without necessarily risking your promotion opportunities), and institutions can lose reputation for hosting big egos who suppress opposing (but ultimately proven correct) views.

[PaulHoule]

Most biomedical papers are rejected out of hand when a metaanalysis is done because of various reasons they are invalid.

These should not be published or funded.

For instance there was a study that found COVID-19 patients had little or no detectable Vitamin C in their blood. Turns out Vitamin C levels vary greatly in populations depending on what they eat, if they smoke tobacco, etc. Also the Vitamin C test requires a very low temperature cold chain and it is unusual so I wouldn’t expect a lab that never did it before to do it right.

If they had a control there would still be problems (would the sampling procedure really be the same in a ‘wartime’ COVID ward as the controls?) but many of the concerns above would be addressed. As it is the result was worthless and should have been rejected.

[bluish29]

It is hard to find alternative form of peer review but not impossible. The harder is that once it ia found will be convince people in scientific communities to adapt.

In many fields nowdays ( particle physics and machine learning for example) it is standard and norm to publiah to Arxiv with the understanding that this is pre-reviewd platform for acceasing latest ideas more than latest research. Grad students, postdocs and professors all understand that.

But usually there is requirements from fuding agencies to publish in traditional journals so you have to do it even it is not required logically.

To illustrate that, lets consider a paper about latest results from ATLAS or CMS experiments at CERN. There are potentially at least internal review of a couple hundred physicists participated in discussion and authorship not to mention the period of public comments (there are thousands of people work in the experiment) after that you would apply to publish in physical review where the editor will assgn two or more reviewers to review that?. Usually there is not much comments but sometime happens but results usually gets announced before this and normally the published version would be important for citation purposes.

I think the dynamics of each field would play an important rule of what can we change about current model. But peer review as a theoretical idea is important but not how is implemented. The issue lies also with the whole scientific publishing model that is controlled mostly by for-profit companies with peer review process being free service.

Search on Twitter for example and you will find many complaints from reviewers that they get two or three papers and being asked to review them and provide a report over a weekend or a very tight schedule in general while doing this for free.

Changing the model would require pressure on funding agencies and publishing journals. Academics who rely on publications for promoted and being number one qualifications in most of benefits or eveb prestige are not suitable to lead this pressure at least until public pressure is applied on universities and research facilities to tackle this problem.

[Hans_Offman]

Find a way to "Wikipedia" or "blockchain" the concept of peer review. If one were able to get actual experts in the field to "compete" for the truth on the individual points within the paper then one may have something promising. Also, look into the concept of having failed attempts included in the proof manifesto - "hey, we tried to get a new result, but our attempts only yielded this". Otherwise, people who submit profound paper's may be left dealing with some TAs who rejects their paper due to two spaces being after a period, or not using the latest vernacular of the field in question.

[Jensson]

Why don't we have public peer reviews? Post the paper publicly and anyone can comment about the validity of the paper. Then if it gets accepted it gets the stamp of approval of the "journal" to publish it.

Why don't we do this already? Probably because the "journals" still calls themselves publishers, so posting the paper before it is accepted would be seen as publishing, and they want to "publish" it first. But scientific journals are no longer valuable as publishers, just for their stamp of approval, the idea that they need to be publishers is holding science back.