
What it was like to be peer reviewed in the 1860s - sohkamyung
http://physicstoday.scitation.org/do/10.1063/PT.5.9098/full/
======
arcanus
I'm not a fan of returning to this form of communication.

In a recent review, which was single-blind, I had a paper from a group I know
well and the principle author was a heavy-weight in my field, and certainly
will review proposals I write in the future. The paper had good content but it
was clearly written by a graduate student of his and it really needed work. I
made numerous suggestions for revision that I would have felt very hesitant to
do if the reviews were fully open (e.g. if my name had been attached). I would
have been inclined to simply accept this paper with minor revisions so that
the PI did not have any impression I was being unreasonable.

As another example, I had a recent double-open review for a conference where I
had to attach my name to a review that was not able to accept the paper. I
felt (and still do) that this submission was absolutely _not_ acceptable given
the standards of this proceedings. Now, this group has seen my name and
associated it with a 'reject', and I am worried that should we meet in the
future, this could cloud our interactions.

For both, I've thought about this a great deal and I stand completely by my
reviews, my criticisms, and I did try to come across as professional as
possible. But I would have preferred to be anonymous, and I see no reason my
name being attached would have helped the review process.

~~~
santaclaus
It's not like you can't get a 95% idea of who is reviewing a paper now, at
least in CS. Look up the list of technical committee members (since CS is
conference centric), find the two with enough expertise in your area to be the
primary or secondary reviewer, boom you have found two of your reviewers.

Typically works when the papers are blind too, people have very different
paper writing styles.

~~~
arcanus
Good point. For the first paper, I bet that group would have put me in a top-5
list, and they certainly would have put my group (or someone from it) in the
first place.

But that slightly anonymous element helps, because typically you aren't _sure_
who reviewed your paper.

------
BeetleB
Peer review, as it is known today, only took off at a large scale with the
invention of the photocopier.

There's a story where Einstein _withdrew_ a paper of his after he found out
the editor had sent it for peer review without his permission. He was offended
- it just wasn't the norm back then.

[http://michaelnielsen.org/blog/three-myths-about-
scientific-...](http://michaelnielsen.org/blog/three-myths-about-scientific-
peer-review/)

~~~
nonbel
Yes, when I looked into it I found there is actually zero evidence that
institutionalized peer review is helpful at all. In fact, it looked like it's
primary role was to impede science by enforcing whatever prevailing bias
exists at the moment. For example:

>"In addition to being poor at detecting gross defects and almost useless for
detecting fraud it is slow, expensive, profligate of academic time, highly
subjective, something of a lottery, prone to bias, and easily abused."
[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/)

~~~
JohnStrange
I've met many critiques of the peer-review system (including myself), but have
never heard a compelling suggestion with what to replace it and why the
replacement would be better. Without peer review, editors would have to assign
papers by gut feeling or perceived reputation of the author, or the scientific
community would be flooded with massive amounts of papers that are way below
acceptable quality or even entirely false, not the speak of the dangers of
pseudo-science and fake scientists or even paid shills.

However, I believe open access journals can and should experiment with
alternative forms of peer review. For example, forcing every submitter to
anonymously score 5 papers before his/her own submission is accepted, a karma
system, a good meta-moderation system, and a good document revision system
taken together could work for online publications. There is a lot of potential
for innovation in online publishing.

The problems are in the end similar to those on online forums and social
media. It's hard to keep trolls and people who try to game the system away and
equally hard to prevent people from scoring solely according to their personal
tastes and with kind of 'political' (as in politics of science) motivations,
especially when the process is anonymous. But ordinary peer review has the
same problem.

~~~
nonbel
>"have never heard a compelling suggestion with what to replace it and why the
replacement would be better"

You can't just replace/remove peer review and expect things to improve. It is
part of the whole "assembly line" science environment that was created when
the US government became the primary source of funding in the 1940s or so.

If we want functioning science, we need to replace at least two things:

1) The search for "significant" p-values with comparing precise predictions to
data.

2) Peer review with independent replication.

Both are pretty much just reverting to how things worked earlier.

~~~
BeetleB
>The search for "significant" p-values with comparing precise predictions to
data.

Problems:

1\. Not sure we have a decent alternative.

2\. While p-values are heavily used in some fields (social sciences, medicine,
etc) they are not in others (e.g. physics). Yet those other disciplines still
have similar problems.

~~~
nonbel
What do you mean about not having a decent alternative? The perfect
alternative is given in the second part of what you quoted.

Also, I see that physics is now starting to rely on statistical significance
(eg LIGO), which I guarantee will be to their detriment. They are just behind
those other fields in adopting it.

