

Rethinking the scientific method - neilc
http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer

======
powera
Considering that most of these studies are by people with an interest in
getting a certain result, it's likely experimental errors/not publishing some
results will bias the published numbers in a certain direction.

The scientific method has never been proof against people that lie by
omission.

~~~
gruseom
He's writing about results that were validated through replication. That _is_
the scientific method. If what he's saying is accurate, the problem isn't a
few bad apples (that's the proverbial reflexive defense of the status quo
anyway).

 _But now all sorts of well-established, multiply confirmed findings have
started to look increasingly uncertain._

I have no idea if it's accurate, but Lehrer's a credible journalist, less
prone to the sensational than most. If he's writing about this, it's probably
because serious scientists are concerned about it.

(On another note, when I pasted the above, the following text showed up
appended to my selection: _Read
more[http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_...](http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz17PM5acPA)
_ How can such a venerable publication as the _New Yorker_ resort to such
tackiness?)

~~~
bugsy
The replication they are talking about in these particular pharma studies are
of the nature that a pharma corp orders up twelve identical 8 person studies.
Three studies come back showing their new pill helped a tiny bit, six show no
the pill did nothing, and three show the patients got worse. They then cherry
pick the three that showed it helped, and possibly toss in one of the ones
that showed the pill did nothing just to cover up what they are doing. They
then publish a paper showing that 3 out of 4 studies validated that the pills
work. The other 8 studies are set on fire and never mentioned. And there is
your multiple studies.

This is not some crazed conspiracy theory either, this (doing multiple very
small sample size studies rather than one slightly larger study and discarding
and never mentioning some of the studies with the least favorable results) is
actually how it is now known to be done, and this methodology is the reason
why some journals have started to say that all of the studies have to be
registered in advance if they want to publish their results so that the
companies can't selectively discard results like they have been doing.

~~~
drndown2007
A great article about how shaking our medical science is:

[http://www.theatlantic.com/magazine/print/2010/11/lies-
damne...](http://www.theatlantic.com/magazine/print/2010/11/lies-damned-lies-
and-medical-science/8269)

And yet, surgeons and medicines have definitely helped me....

------
defen
Is there any non-paywall way to view the article? This looks like it might be
interesting but I can only see the first couple paragraphs. The title makes me
think of three things: Paul Feyerabend, Bayesian inference, and the increasing
effectiveness of placebos. I'd be curious to see if any of them are in the
article.

~~~
michael_dorfman
I don't know of any non-paywall way to view the article; I'm a (satisfied) New
Yorker subscriber, so I've already paid.

It's a great article, by the way.

The answer to your latter (implied) question is "no"-- the article does not
explicitly mention any of those three things.

------
vacri
A rather sensationalist headline given that the scientific method already
recognises the problems of human researchers and not only offers some
mechanisms to counter those problems, but is eager for more.

~~~
rickmode
I agree. The article's headlines has a distinct anti-intellectual flair, and
yet the article itself points out human flaws in research rather than a
fundamental flaw in the scientific method.

------
swombat
This is not so much a problem with the scientific method as a problem with
self-interested applications of statistics. F still equals m*a hundreds of
years later, and that's not about to change.

~~~
gruseom
Yet I wonder if "the" scientific method has in fact been overapplied way past
its sweet spot of mechanics and so on. It's by no means obvious that it works
as well in, say, medicine, let alone psychology, let alone sociology, let
alone economics. The fact that it's the cult of our age perhaps blinds us from
asking the interesting questions about its limits.

~~~
JoachimSchipper
Why?

Medical science seems, sad to say, incompetent
(<http://care.diabetesjournals.org/content/17/2/152.abstract> rediscovers
integration, and statistical illiteracy among doctors has been repeatedly
reported) and corrupt (conferences in lavish locations, vendor-sponsored
efficacy research).

I'm sure there are good researchers, don't get me wrong, but I'm also not
surprised that some really questionable stuff gets reported.

I'm sadly, even less aware of the state of research in sociology, but I'd like
to point out that non-scientifically-tested psychological theories tend to be
complete nonsense (cf. lots of Freud's work). With respect to economics: the
field has many problems, but a book like Freakonomics is a very readable
argument in favour of the use of statistics in (micro-)economical science and
related fields. (Do note that the scientist-author, Levitt, actually
understands statistics. This is, I suspect, important.)

That said, there _are_ issues with the scientific method, like the fact that
it tells us to drop falsified hypotheses but does not, in itself, teach us how
to find interesting results. That's not germane to your observation, though.

~~~
khafra
One reason the scientific method does not work on economics is that markets
are anti-inductive[1]. If you can establish a regular pattern in their
behavior, market participants start exploiting that pattern, and it goes away.

[1] <http://lesswrong.com/lw/yv/markets_are_antiinductive/>

~~~
JoachimSchipper
I'm not sure this is such a strong attack on the scientific method. First,
there's _much_ more to economics than predicting the stock market - a proper
understanding of demand curves is useful even if everyone else has it as well.
In fact, I'm not sure that predicting the stock market has more to do with
mathematics than with mass psychology.

That said, I _do_ agree that properties of the stock market, once widely
known, tend to be arbitraged away. From the article:

> There was a time when the Dow systematically tended to drop on Friday and
> rise on Monday, and once this was noticed and published, the effect went
> away.

However, that doesn't mean that the scientific method cannot produce true
conclusions - it's just that those don't tend to _stay_ true.

lesswrong has much more impressive criticisms of the scientific method, IMHO.
(The main one being that, while the scientific method will hopefully prevent
you from endlessly clinging to a wrong hypothesis, it won't tell you how to
find _good_ hypotheses.)

------
crayz
Here's a PDF of the full article: <http://crayz.org/science.pdf>

~~~
JoachimSchipper
It's interesting and all, but I think the New Yorker has a right to restrict
access to paying subscribers.

EDIT: well, non-paying subscribers apparently. That makes me feel better.

~~~
crayz
I downloaded it by signing up for a free account, and it's littered with their
ads

------
bugsy
The scientific method is fine. Note that this is about certain psychiatric
pharmaceuticals being found not to have the efficacy that was earlier claimed,
in studies done by pharmaceutical companies who doctored results and played
statistics games and selectively hid results that didn't support
profitability.

It's fascinating to see the level of hubris in the corporate pharmaceutical
industry, that after cornered and confronted with the fact that for decades
their self-serving fabricated so called research is fraudulent, they would now
have a conference to announce that the scientific method has failed.

~~~
michael_dorfman
Did you read the article? It is not primarily about the pharmaceutical
industry, or about doctoring results, although it touches on both of those
topics. The argument is far more subtle, and more far-reaching.

~~~
drndown2007
Agreed. The one that has me wondering how far off we might be is the idea that
radioactive decay might not be constant:

[http://io9.com/5619954/the-sun-is-changing-the-rate-of-
radio...](http://io9.com/5619954/the-sun-is-changing-the-rate-of-radioactive-
decay-and-breaking-the-rules-of-chemistry)

~~~
bugsy
That too demonstrates the scientific method in action. Experiments showed that
the prevailing paradigm is likely flawed. Now we know more.

