

A Statistical Test Gets Its Closeup - clay
http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-closeup-1050/

======
ramanujan
Great quote in the WSJ comment section:

[http://blogs.wsj.com/numbersguy/a-statistical-test-gets-
its-...](http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-
closeup-1050/tab/comments/)

    
    
      The agency will bend their statistical significance
      requirements when hypothesizing that a treatment has
      risks, but almost never does when considering the 
      efficacy of a treatment.
    

That is the long and short of it. The FDA makes up the rules as they go along.
As for the particular issue at stake, the Supreme Court's ruling in the Zicam
case is horrifying:

[http://online.wsj.com/article/SB1000142405274870371250457623...](http://online.wsj.com/article/SB10001424052748703712504576235683249040812.html?mod=googlenews_wsj)

It is indeed true that statistical significance is not the be-all and end-all.
It is also true that you need _some_ method of determining whether a result
could be due to chance or not.

This is particularly true when you have trial lawyers and regulators who smell
blood. Dow Corning was sued and had its reputation dragged through the mud
because people thought breast implants were causing some kind of disease.
Turns out that wasn't true at all:

[http://query.nytimes.com/gst/fullpage.html?sec=health&re...](http://query.nytimes.com/gst/fullpage.html?sec=health&res=9B03E6D9103BF932A15755C0A96F958260&n=Top%2fReference%2fTimes%20Topics%2fOrganizations%2fI%2fInstitute%20of%20Medicine)

Now that statistical significance has been jettisoned as a defense, any group
of anecdotal injuries of dubious provenance is in theory grounds for a class
action lawsuit. And trial lawyers specialize in turning that particular kind
of theory into practice. There's a reason that "mesothelioma" was the #1
Adword for a while -- ambulance chasers bid on it like crazy.

The Supreme Court has just made it a whole lot easier for them.

------
crasshopper
For the non-Bayesians, here's a paper that explains in detail one of the
strongest criticisms of null hypothesis testing, by bayesian John Kruschke:

[http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.p...](http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.pdf)

To summarize, an experimenter's internal mental state (intentions) can affect
the t statistic. That is: not changing the experiment at all, but just
thinking something different -- gives a different level of significance.

In Kruschke's words: voo-doo.

