
How to constructively review a research paper - ingve
https://freedom-to-tinker.com/2018/05/15/how-to-constructively-review-a-research-paper/
======
gervase
This article reads as more of a "My idealized view of how to constructively
review a research paper", rather than a concrete foundation to evaluate
papers.

From the reviews my publications have received, it's quite clear that almost
every reviewer (including myself!) has constructed their own ad-hoc mental
criteria of what they're looking for in a paper. This encourages the evolution
of highly-specialized, ultra-niche subdomains and conferences, where one wrong
word can disqualify you as a "bad venue fit", and when you're writing your
paper, you're trying to predict what the reviewers will think more than about
the technical information you're presenting.

If a more objective, standardized review process (as implied by the article
title) could be created, even if it was only applied at the meta-review level,
I think this could provide some benefits.

Of course, that would require everyone to throw away their own mental models
and learn a new system, so I think it's highly unlikely. But it's nice to
imagine, isn't it?

~~~
btrettel
> If a more objective, standardized review process (as implied by the article
> title) could be created, even if it was only applied at the meta-review
> level, I think this could provide some benefits.

I'm on board with this.

My subfield in particular could use some basic standardization in terms of
sanity checks on claimed results. It's always irritating to invest time to
read a paper, only to later realize the authors make a false claim that could
have been caught with a simple counterexample. These sorts of errors can
easily be caught and it would save everyone time if they were caught at review
rather than later.

(There's one case in particular that I have in mind. The result seemed
plausible on its face, but one can generate multiple counterexamples. The
actual error was subtle in the original paper. I'm half convinced it wasn't
caught because the original paper was written in a way that obscured the
error, certainly unintentionally. I'm currently waiting for my paper
addressing this error to be reviewed.)

