

Who's Afraid of Peer Review? - c0rtex
http://www.sciencemag.org/content/342/6154/60.full

======
makomk
See also:
[https://news.ycombinator.com/item?id=6491226](https://news.ycombinator.com/item?id=6491226)

Edit: This _looks_ remarkably like an attempt by Science to smear their open-
access competitors, given that it didn't even bother to test whether non-open-
access journals have the same flaws (and in fact we know they do, including
Science!), yet it's being presented as evidence that open access journals
specifically are flawed. I'd hope this article would never survive peer
review.

~~~
ggchappell
> This looks remarkably like an attempt by Science to smear their open-access
> competitors, ....

That may be. I wish this had been done differently, and I wish "open access"
were not the term used. It may be true that the author is trying to paint with
an overly broad brush, throwing some FUD at the movement to make scientific
papers easier to get at; I'm not sure.

But it is clear to me that this article is describing a real problem. The
problem is the recent proliferation of new journals with very low standards.

> ... and in fact we know they do, including Science! ....

I don't agree. Problems with the peer review system have been demonstrated. It
is inconsistent, sometimes lazily done, and subject to corruption in some
fields.

But if someone were to claim that you could take 300 journals of the caliber
of _Science_ , send them all a worthless paper, and get acceptances from half,
then I would call that ridiculous. The occasional acceptance of a poor paper
means that the review system needs work. The acceptance of poor papers as a
matter of course indicates a lack of standards. There is a difference.

~~~
mckoss
I think the point of the comment is that Science's study is purely self-
serving, since they did not also send their sting paper to the more
traditional journals to test their peer review rejection rate of this bogus
paper.

------
ggchappell
A point needs to be brought out here. This article is not about traditional
academic journals. It concerns journals of a kind that have been popping up
like mushrooms in recent years. The article refers to them as "open access".
I'm not fond of the term, since there are certainly selective, reliable
journals that still allow submissions from anyone and publish articles on the
web without a paywall. Regardless, plenty of more-or-less fake journals have
been founded recently. As a researcher, I get solicitation e-mails from them
every day.

It is true that it can be difficult for someone not in the relevant field to
tell the difference between different kinds of journals. I know the
difference, since I've heard of the major journals in my field, I know many of
the editors, etc. But a science journalist might have more trouble with it.

For fields like medicine, this can matter a lot. It's the same problem we see
with SEO and the like: how, in the modern world, do we form a robust measure
of reputation and reliability?

~~~
mturmon
Well put. I think _Science_ does have some conflict of interest here, but I
also think they are one of the good guys in the publishing business, and that
this is a worthwhile subject for an expose.

Like you, I get (about 1/week) blind offers to submit articles to this or that
journal, and even (at a lesser frequency) offers to be an editor (unpaid!). It
stinks to high heaven [1]

Getting back to _Science_ , their individual subscription rates are reasonable
(~$150/year = 52 issues), their public health articles are available
immediately after publication, and their other articles are available after 1
year. They are run by a non-profit institution (AAAS), so they are less
motivated to jack up their fees.

I was not able to find institutional fees for _Science_.

Incidentally, the situation with _Nature_ is different -- they are for-profit
([http://en.wikipedia.org/wiki/Nature_Publishing_Group](http://en.wikipedia.org/wiki/Nature_Publishing_Group))
and it seems to show in their rates: [http://today.ucla.edu/portal/ut/uc-
libraries-faculty-protest...](http://today.ucla.edu/portal/ut/uc-libraries-
faculty-protest-proposed-160117.aspx)

[1] Here are the most recent 5: "International Transaction of Electrical and
Computer Engineers System" (sounds like a joke), "Entropy", "The Open Software
Engineering Journal", "Journal of Robotics", "The Open Electrical & Electronic
Engineering Journal".

------
jvanderbot
Peer review is receiving a bad review, no pun intended. Please remember that
good journals exist, and insiders know good from bad. A bad acceptance from a
heretofore-unknown journal with a seeming credible title is not indicative
that all peer review is flawed or done by impatient baboons.

And I say this having had a terrible time getting a decent review in recent
conferences. But my journal review process has been largely positive, helping
to improve my paper significantly, even if they miss some things. Certainly,
when I receive a paper to review, I and my colleagues give it our best shot,
but we largely work only for the established upper-echelon in our field.

~~~
jnazario
"insiders know good from bad"

while often true, science and scientific journals do not serve solely the
scientific community, they serve humanity, which includes the non-professional
public. while you may be an honest person and would never knowingly abuse
wrong information to harm others, gain, or advance a broken agenda, plenty of
people do. the "vaccines lead to autism" people, for example, do that.

the history of science shows that careful, accurate and correctly performed
experiments and publications are what give humanity progress. their absence,
history has again shown, causes humanity to retreat, sometimes with disastrous
consequences.

this is not an idle, "i sure wish more 'scientists' could publish" debate, at
some level the very question of human progress is at stake.

------
Fomite
Major Comments: The author make a compelling case that a sizable number of
Open Access publishers have potentially flawed review policies that allow
substandard work to enter the literature. This is a promising first finding,
however the authors then use this data to build a narrative of the nature of
Open Access journals without having performed the corresponding experiment
with an appropriate control arm.

Without a comparison to conventional publishing, these findings cannot be
extended as far as the author does in the conclusions. Such an assertion,
without support, is not justified or supported.

Recommendation: Resubmit after Major Revision.

------
army
This isn't that surprising. It's common knowledge in the academic community
that:

* There are bogus open access journals which exist solely to make money from author fees.

* There are bogus conferences that follow a similar business strategy.

* There are lower-quality journals and conferences that do not attract many top-quality submissions, so tend to have a lower barrier for acceptance: you might get work that's rigorous and unexciting, work that's exciting but preliminary or rough around the edges, or work that's neither rigorous nor interesting.

* Peer reviews vary widely in quality, from being thorough to overly critical and nit-picking to being cursory. This is extremely frustrating to many people, but is hardly an undiscussed issue. One particular problem is that more junior reviewers are often reluctant to criticize a paper unless they're very confident in their understanding, either due to it being slightly outside of their core expertise, or due to convoluted presentation: it's easier to assume it's correct than to go out on a limb criticizing something.

I would say that being published in a peer reviewed article is just the start
of a gauntlet that any serious scientific claim has to run, regardless of the
quality of peer review. Plenty of important work has started out as being
extremely controversial: even once it got published people didn't necessarily
think it was right. The consensus does continue to evolve.

I've never met a scientist who will believe something just because it's in a
peer reviewed article.

------
robotresearcher
There is a serious ethical problem with submitting dummy papers in that it
abuses the trust of the editors and reviewers. You are wasting the time of
volunteer reviewers who have nothing to gain but plenty to lose if they screw
up. Reviewers assume papers are submitted in good faith, and are not traps.

Editors have practice spotting cranks but trap papers don't have that smell
about them since they are written by smart but naughty people trying to trick
reviewers.

~~~
betterunix
If nobody ever _tests_ the review process, how can we know that it is working?

~~~
robotresearcher
We read the journals. Many readers of an article are qualified to review it.
Many also personally know the editor or associate editors and will raise a
stink if the journal starts to suck. We _are_ these journals. We write the
articles for free, we review them for free, we proudly share our results. We
hate it when a shitty paper gets in.

Edit: There _are_ shitty journals that no one reads. Everyone in the business
knows which ones these are. There is a problem explaining this to lay people,
since the shitty journal web site doesn't have a sticker on it saying "zero
credibility - please ignore these articles". This is not really a problem for
researchers.

~~~
betterunix
That is not relevant to the issue of whether or not the peer review process
works in deciding what articles to publish. Researchers might be able to
identify bad journals and bad articles (though in my experience, quite a few
researchers only spend time reading the results), but the point of the review
process is to ensure that bad articles are not published in the first place.
If the reviewers are allowing bad articles to slip through the cracks, then
the process is not working and we need to seriously reconsider the entire
journal system (though I would say that the Internet already requires us to
reconsider that system).

My real point was that submitting blatantly bad articles serves a purpose,
which is to test whether the current process of reviewing articles is
effective at weeding out bad articles. We should not blindly assume that
journals, even top-tier journals, are only publishing good articles.

~~~
robotresearcher
I addressed this point in the third sentence: "[Researchers] will raise a
stink if the journal starts to suck."

------
cup
>Whos afraid of peer review?

Anyone whos ever tried to publish. I used to love science before I actually
became an academic and realised how dark and corrupt it could be behind the
scenes, I suppose the same stands for every field though.

~~~
davorak
> I suppose the same stands for every field though.

That is not my understanding/observation, some fields seem to be considerably
worse then others, while some seem to have relatively little.

