

Three myths about scientific peer review - neilc
http://michaelnielsen.org/blog/?p=531

======
cperciva
While I agree with the author's main points about peer review, I think it's a
bit of a stretch to conclude from those that peer review is useless.

As a peer reviewer, I've sent papers back with comments along the lines of
"this work seems quite similar to the published work of X in the following
papers (which the author doesn't cite): ... I wonder if he if the author is
aware of this earlier work", and subsequently heard nothing more (presumably
because, upon looking at said papers, the author came to the conclusion that
he had not in fact discovered anything new). On the opposite side of the coin,
one of my papers benefited from very similar comments, in that having related
work brought to my attention allowed me to revise my paper to better describe
how it fit with the rest of the field.

I think it's in cases like this -- where peer reviewers point out things
(whether other research, or possible errors) of which the author(s) do not
seem to be aware -- that peer review really shines. If reviewers respond
merely with a 'yea' or 'nay' they're not doing their job properly; their job
is to _review_ , not to _judge_.

~~~
yummyfajitas
The quality of peer review varies with the journal. For instance, Journal of
Computational Physics tends to give out helpful reviews. From JCP, I usually
get back a list of helpful suggestions and weak spots in the paper. Even the
incorrect suggestions/criticisms tend to reveal ambiguity in my writing more
than anything else.

PRL, not so much. In my experience, at least one referee doesn't even finish
reading the paper. It's not just me, this happened to a friend of mine last
week:

Referee: "The authors do not demonstrate that their equations apply to any
physical example."

Response: "See page 3 paragraph 4."

------
jderick
I don't think anyone who has ever reviewed a paper would consider these three
things to be "myths" -- they are just obvious facts.

I do think the peer review system will eventually change to something more
like hacker news. It is a much more convenient way to do things than trying to
find volunteers, mail out papers, collect responses, etc. Not to mention you
can potentially draw from a larger audience and you make your work more easily
accessible.

One mechanism that does exist in peer review that I'm not sure could be easily
reproduced is a sense of duty while reviewing a paper. When selected to review
a paper, you know you are one of only a few who will determine the fate of
this paper and you feel a responsibility to try hard to understand the paper
and give it a fair critique. If a paper were just posted online for anyone to
critique, people might not put forth as much effort in their reviews.

------
aposteriori
Some people say double-blind reviews make things better since reviewers aren't
(theoretically) influenced by the identity of the authors, and authors can't
tailor their paper to suit the reviewers (or try to get back at them later for
negative reviews).

How about completely open reviewing, like in some of the BiomedCentral medical
journals (e.g., <http://www.biomedcentral.com/bmcmed>)?

If done properly, all the cards are on the table. A paper cannot (should not)
be rejected or accepted without legitimate reasons. Reviewers and authors can
be made to account (through tarnished reputation and risk of embarrassment, at
least) for their actions.

