
When does peer review make no sense? - nkurz
http://andrewgelman.com/2016/02/01/peer-review-make-no-damn-sense/
======
eanzenberg
When did p < 0.05 become acceptable? I remember in astronomy/astrophysics
where 5sigma (3x10^-7 probability) was always the minimum required to publish.

~~~
Amorymeltzer
0.05 was first proposed by Fisher himself, which is 2 sigma if you look at a
normal distribution (95% falls within 2 standard deviations[1]). Indeed,
physics (generally particle physics, such as the Higgs Boson) goes for 5 or 6
sigma.

In biology, 0.05 is used because, well, it's hard otherwise. The cost, time,
effort required for plants, animals, and even cells get exorbitant quickly,
and are subject to many, many variations (animals generally have additional
IACUC limits on procedures and numbers).

I could make arguments about how particles obey laws while biology is at least
a few levels abstracted over that fundamental behavior, but xkcd has already
done that. I personally feel that p<0.05 is absurd, and the entire biological
field NEEDS to move to p<0.01. There's a movement toward that end, and it's
about damn time.[3]

1:
[https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rul...](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule)

2: [https://xkcd.com/435/](https://xkcd.com/435/) &
[https://xkcd.com/1052/](https://xkcd.com/1052/) &
[https://xkcd.com/1520/](https://xkcd.com/1520/) spring to mind

3: A lot of internet ink has been spilled on the topic, but
[http://www.nature.com/nmeth/journal/v12/n3/full/nmeth.3288.h...](http://www.nature.com/nmeth/journal/v12/n3/full/nmeth.3288.html)
&
[http://www.pnas.org/content/110/48/19313.long](http://www.pnas.org/content/110/48/19313.long)
& [http://www.nature.com/news/scientific-method-statistical-
err...](http://www.nature.com/news/scientific-method-statistical-
errors-1.14700) are good places to start.

------
leephillips
Another situation where peer review makes no sense is sometimes recognized by
traditional peer reviewed journals. _Physical Review Letters_ has suspended
peer review on occasion for high-energy physics experiments with thousands of
authors, acknowledging that it would be difficult to find a competent reviewer
who is not already on the author list.

------
exelius
So to answer the author's question; peer review makes no sense when your peers
are incapable of judging a paper's validity either through a lack of domain
knowledge on the subject or an overall poor grasp of scientific concepts in
the community as a whole.

------
norswap
I believe peer review to be generally harmful. It forces you to do things in a
certain direction, in a certain way.

To quote Knuth:

"In fact what I would like to see is thousands of computer scientists let
loose to do whatever they want. That's what really advances the field."

And today, the idea marketplace does not need to be mediated by conferences
and papers.

There is however, one aspect in which removing them is tricky, and that is
funding. Currently, publication is a metric used to grant funds or evaluate
the result of research. If we removed that metric, it would just likely be
replaced by other forms of gate-keeping that are as bad. And giving money to
anyone is kind of problematic, of course; even just a permissive filter
(checking for "reasonableness") would let in way too much candidates w.r.t.
available money. I don't really know what could solve that particular
conondrum.

~~~
tensor
This suggestion is as extreme and misguided as suggesting that code review is
harmful.

~~~
norswap
This suggestion is as extreme and misguided as suggesting that this suggestion
is harmful.

(This is not a typo :] )

