

Peer Review is No Panacea - cwan
http://www.theatlantic.com/business/archive/2010/06/peer-review-is-no-panacea/58842/

======
kurtosis
I would Love it if referee reports were published with papers. Why waste all
of the unpaid time and research of the referees by compressing their opinions
into a yes/no/revise decision? This would improve the quality of the reports,
and make the process of reading papers much easier. One of the most difficult
parts of reading a paper is trying to decide for yourself if the methods or
conclusions of the paper are flawed. The referee reports are a valuable
counterpoint from people who probably know more about a topic than you do!

~~~
albertcardona
The Journal of Brain Research does that, and did so for decades. But there is
a bias: only papers with good reviews are published. What happens to good
papers with bad reviews is that they don't get published, or get published
elsewhere.

------
pradocchia
In my personal experience, peer review follows fashion, and fashion is
generational. Couch your results in the fashion of the day, cross your t's,
dot your i's, and you're all set. Publish frequently, in innocuous little
bites. Wait for the old guard to retire before publishing the synthesis that
brings your prior publications together into a new and coherent whole.

Now you're the old guard. Repeat.

------
bonsaitree
Summary: Imperfect system is imperfect.

'Ya gotta love a tautology adorned as journalism.

Are there any reasonable alternatives for privately funded research? No. There
will always be the urge to game data to fit the hypothesis and honest people
will still make mistakes. This forever shall remain a challenge for science.

I'm struggling with the urge to flag, but recognize the applicability to HN.

~~~
jerf
I fail to see your point. The core claim that there are people who put far too
much stock in the words-treated-as-magic "peer review" is definitely true; I
encounter them all the time, to the point that I've also come to recognize it
as a pattern. The fact that this is not true is, as you say, tautological. But
some people need this fact brought to their attention, as it is clearly not
generally understood.

I've come to classify attitudes about science now into three broad categories.
Hostilty: "I don't care what science says, Homeopathy worked for _me_."
Religious: "Peer review is never wrong, and therefore the consensus is also
always correct in every particular and those who disagree are _heretics of
science!_ " Realistic: "Science is the best way we have to find truth, but it
isn't perfect either. The history of science should be learned and carefully
examined, and it should not be forgotten that everything we see in the past is
still around today, the only question is _where_." Rather a lot of people fit
into that second one, and think it is the best understanding of science there
is, and tend to label people with the third level of understanding as heretics
on par with homeopaths, too. Trying to prod people out of religion-of-science
into realistic-understanding-of-science is a noble goal, worthy of a blog
post.

Science has been wrong before, for decades at a time in every discipline I've
ever studied enough to learn about its history (in addition to the current
consensus), and the question is not _whether_ there are disciplines every bit
as wrong right now, but _which ones they are_. If you don't understand that or
haven't internalized it, you _don't_ understand the process of science, even
if you can recite its tenants at will. When we say that science is a process,
not an end-result, that isn't just words, it is the reality, and what it means
in practice is that the consensus is very frequently wrong. There's nothing to
"correct" in consensus if it's presumed always correct. The religious idea of
science _makes no sense_ ; consensus _must_ be wrong at some points in time,
it is a _necessary step_.

(This is all discipline-neutral, by the way. I do have my opinions about which
disciplines are most likely to be wrong right now, but I freely acknowledge
those are _opinions_. Certainly in the past entire disciplines have made
glaringly-obvious-in-hindsight-and-common-sense mistakes before and it is no
great insight to think that there might still be such errors around. But
again, I freely admit that only time will really tell, and there are certainly
other wrong-consensus-views I agree with.)

~~~
dantheman
I also think a big part is how forthright a scientist is. Most scientists
claim way more than can be supported by their evidence, and it's not always
clear what assumptions are being made. There's a great talk, I can try to find
it if there is interest, that discuss the hubris that most climate scientists
have. Observing a system for a tiny portion of a cycle can in no way tell you
the entire cycle. If you then admit that "hey we don't know, but here's our
best guess, and here's why we think it" then you've got something. Once
something is established like like copper conducts electricity or the speed of
light then you can say "hey we're pretty sure about this". It has a lot to do
with the maturity of the science and the tools and techniques available to the
scientists. Complex systems that cannot be decomposed and upon which
experiments can not be made have should have a much lower confidence value
than repeatable tried and true experiments. To conflate them is to cause
confusion and to weaken the trust in more established science. Peer review is
more about how to position your work in relation to others for ease of
understanding than to verify the results independently.

~~~
thaumaturgy
> _Most scientists claim way more than can be supported by their evidence..._

Ironically, you've made a claim here which is unsupported by any evidence at
all. ;-)

Seriously though, the thing about this is that research and experimental
scientists are working in environments that you and I know very little about.
Their claims are an extension of their experience in that environment; they
spend a great deal of time for example designing experiments or attempting to
reconcile large volumes of data, and when they've spent months -- or, more
likely, years -- doing that, they might publish a paper.

Then the public occasionally gets ahold of this brief summary of the
scientist's work over that period, and they object that the claims are
specious or sparse or incomplete.

This is not to say that a scientist's claims should be trusted just because
they're a scientist, but rather that the only acceptable challenge to a
shortcoming in a scientific presentation is ... more science, not argument or
conjecture.

To put it another way: it is extremely unlikely that a layman will find a
consequential error in a peer-reviewed scientific paper published in a
reasonable journal, especially without any domain knowledge in that particular
field.

~~~
dantheman
Errors in science are well known and are part of the process, I didn't think I
needed to provide evidence of that. Unfortunately science that deals with
complex systems is much more difficult than those that don't. For instance,
just look at nutrition research, look at the history of the food pyramid and
you see poorly understood science having a substantial impact on those who are
not experts because the nutrition experts were scientists, and physics works
well so nutrition must also be well understood.

I just want people to bend over backwards to tell me the ways in which they
might be wrong, this is useful because it helps you understand the limits of
understanding and can allow people to make informed decisions.

~~~
thaumaturgy
I wasn't being hostile (this time), I was just having a little fun.

What you're grappling with here is basically the "I don't know what I don't
know" problem. Other people on HN have talked about this before, but
basically, there's the knowledge you know, the knowledge you know you don't
know, and the knowledge that you don't know you don't know. How can you
possibly begin to challenge an assertion in a scientific paper if you don't
even know where the mistakes might be?

(I'm going through something similar right now in a criminal case. I have no
background in law. I do not trust the attorneys involved. It's maddening.)

I agree that it would be nice if as a matter of habit there were a _brief_
statement in papers and presentations outlining incomplete areas of the
research or areas justifying further study. However, I would not want that to
evolve into an onus on the part of the working scientists to educate every
layman with a passing interest in their research.

If you really want to challenge something in a specific field, the best thing
you can do is immerse yourself in that field for as long as it takes to
develop an understanding of the basic principles. Demanding much else is
merely intellectual laziness.

~~~
dantheman
I didn't think you were being hostile, and I hope you don't detect any
hostility from me, none is intended.

There are many incentives to overstate the validness of ones claims and
unfortunately it weakens the overall strength of the field. I'm not asking
that they make it understandable to the layman, but perhaps on any
recommendation they could put confidence intervals on predictions.

Here's a presentation by Carl Wunsch, and here are the slides:
[http://web.mit.edu/esi/symposia/symposium-2009/2009-symposiu...](http://web.mit.edu/esi/symposia/symposium-2009/2009-symposium-
wunsch.pdf)

[http://cdn.static.viddler.com/flash/simple_publisher.swf?key...](http://cdn.static.viddler.com/flash/simple_publisher.swf?key=1d381293&ref=)

He's one of the worlds experts on the ocean, and is saying the same thing i
am.

Here's a feynman quote on it:

 _I would like to add something that's not essential to the science, but
something I kind of believe, which is that you should not fool the layman when
you're talking as a scientist. I am not trying to tell you what to do about
cheating on your wife, or fooling your girlfriend, or something like that,
when you're not trying to be a scientist, but just trying to be an ordinary
human being. We'll leave those problems up to you and your rabbi. I'm talking
about a specific, extra type of integrity that is not lying, but bending over
backwards to show how you are maybe wrong, that you ought to have when acting
as a scientist. And this is our responsibility as scientists, certainly to
other scientists, and I think to laymen._ \-- Richard Feynman, CARGO CULT
SCIENCE

Also to be clear, I'm optimistic and hope we can solve all sorts of problems
and discover how the world works -- but I think to we need to have integrity
and maintain to foster and maintain that trust. The medical field is ripe with
wild extrapolation from small studies, it's a sign of exhuberance -- "Hey,
here's the answer to the problem you having!" When it should be, "It seems
that in some situations this may help, we still have to do more studies and
don't know why it works but it seems to." It's more honest and when things go
wrong, which they almost always do the person understood that it wasn't well
understood.

I encourage you to watch the video - it's by an expert saying that we don't
have enough information to be sure about anything and he proposes the
following advice:

 _In the meantime, study the system; study the control options; but----take
precautions (mitigation), prepare for adaptation, and do nothing that is even
possibly irreversible (which may be almost anything, including of course, the
ongoing injection of greenhouse gases)._

Also his example of gravity waves, I think is extremely enlightening.

~~~
thaumaturgy
I apologize for taking so long to respond -- I needed to get to a point where
I had the time to properly listen to and consider the video. (So I waited
until it was time to shave. Laptops are wonderful. ;-)

I went into the video with an open but critical mind, and attempted to stay
that way even as I realized that he was presenting a point of view which I
would not usually agree with. So, while I think he presented some casually
interesting points, I also find it curious that he seemed to be committing
some of the very same errors which you are criticizing others in the
scientific community for committing, as well as other errors which even I --
as a layman -- was able to catch.

It's entirely possible that his presentation is merely incomplete, and that he
has good reasons for drawing the conclusions that he was presenting, and he
merely couldn't present those reasons to a non-scientific audience. (Which
would be pretty much my side of the argument in this thread, so I'm totally
willing to accept that possibility.)

I won't devote the time to a point-by-point analysis of his presentation,
unless you express enough interest in continuing this to make it worthwhile,
but in short:

\+ He completely -- and carefully -- ignores various geological records which
go back many hundreds of thousands of years, and which have a high confidence
of accuracy in the climatological community;

\+ Those records do present a regular semi-chaotic rhythm reminiscent of a
strange attractor, and our era appears to currently be the zenith of that
rhythm;

\+ And if those records are accurate, then current measurements exceed any
previous maxima of the last many hundreds of thousands of years;

\+ He seems for some reason to draw the analogy that the climate is correctly
modeled as a chaotic system of some sort -- i.e., one which is extremely
sensitive to initial conditions -- even though this is still a matter of
debate within the climatological community;

\+ While the Titanic may _arguably_ be one example where taking no action may
have resulted in a more favorable outcome, coming up with or inventing
countless counter-examples is trivial, so this does not support his case at
all;

\+ His "gambling models" proposal is quite silly, if for no other reason than
that one of the biggest unpredictable parts of the climate are related to
human activity.

A lot of the rest of his presentation seems to be an argument-from-ignorance.
I think he is framing the discussion in such a way as to suggest we have less
information and less knowledge than we actually do, and then within that
frame, he is arguing that we haven't the knowledge to decide what to do.

But I'll think about it some more.

EDIT: Funny, I pulled up a random-ish TED video (searched for "climate"), and
ended up watching a 4-minute presentation which is pretty much exactly the
point I've been trying to make:
[http://www.ted.com/talks/lang/eng/rachel_pike_the_science_be...](http://www.ted.com/talks/lang/eng/rachel_pike_the_science_behind_a_climate_headline.html)

------
stcredzero
"There are No Panaceas" meme: panacea for "[X] is No Panacea" headlines?

</self-referential-irony>

~~~
hugh3
Heh. I searched google news for things that are currently not panaceas:

 _World Cup no panacea for SAFrica's economic woes

Yuan Revalue Is No Panacea for US

Return to Pebble Beach no panacea for Woods_

And a few others I'm too lazy to copy-paste.

On the other hand, "Your search - "is panacea" - did not match any documents."

