This article reads as more of a "My idealized view of how to constructively review a research paper", rather than a concrete foundation to evaluate papers.
From the reviews my publications have received, it's quite clear that almost every reviewer (including myself!) has constructed their own ad-hoc mental criteria of what they're looking for in a paper. This encourages the evolution of highly-specialized, ultra-niche subdomains and conferences, where one wrong word can disqualify you as a "bad venue fit", and when you're writing your paper, you're trying to predict what the reviewers will think more than about the technical information you're presenting.
If a more objective, standardized review process (as implied by the article title) could be created, even if it was only applied at the meta-review level, I think this could provide some benefits.
Of course, that would require everyone to throw away their own mental models and learn a new system, so I think it's highly unlikely. But it's nice to imagine, isn't it?
> If a more objective, standardized review process (as implied by the article title) could be created, even if it was only applied at the meta-review level, I think this could provide some benefits.
I'm on board with this.
My subfield in particular could use some basic standardization in terms of sanity checks on claimed results. It's always irritating to invest time to read a paper, only to later realize the authors make a false claim that could have been caught with a simple counterexample. These sorts of errors can easily be caught and it would save everyone time if they were caught at review rather than later.
(There's one case in particular that I have in mind. The result seemed plausible on its face, but one can generate multiple counterexamples. The actual error was subtle in the original paper. I'm half convinced it wasn't caught because the original paper was written in a way that obscured the error, certainly unintentionally. I'm currently waiting for my paper addressing this error to be reviewed.)
From the reviews my publications have received, it's quite clear that almost every reviewer (including myself!) has constructed their own ad-hoc mental criteria of what they're looking for in a paper. This encourages the evolution of highly-specialized, ultra-niche subdomains and conferences, where one wrong word can disqualify you as a "bad venue fit", and when you're writing your paper, you're trying to predict what the reviewers will think more than about the technical information you're presenting.
I don't have very many paper, but I've started writing ad copy, and this sounds exactly like writing ad copy.
From the reviews my publications have received, it's quite clear that almost every reviewer (including myself!) has constructed their own ad-hoc mental criteria of what they're looking for in a paper. This encourages the evolution of highly-specialized, ultra-niche subdomains and conferences, where one wrong word can disqualify you as a "bad venue fit", and when you're writing your paper, you're trying to predict what the reviewers will think more than about the technical information you're presenting.
If a more objective, standardized review process (as implied by the article title) could be created, even if it was only applied at the meta-review level, I think this could provide some benefits.
Of course, that would require everyone to throw away their own mental models and learn a new system, so I think it's highly unlikely. But it's nice to imagine, isn't it?