
Statistics: P values are just the tip of the iceberg - phreeza
http://www.nature.com/news/statistics-p-values-are-just-the-tip-of-the-iceberg-1.17412?WT.ec_id=NATURE-20150430
======
rm999
In my experience around a lot of professional data analysts, shoddy scientific
design is ubiquitous; I was definitely guilty of it all the time. A mentor
gave me some advice that helped tremendously: formulate what you're trying to
answer _very precisely_ (i.e. the hypothesis), and remember you are only
trying to answer that question during every step of your experiment.

It's very easy to lose focus or rush through important steps that seem
insignificant. This is dangerous because bad methodology often looks a lot
like good methodology but leads to very different results. When you have a
precise hypothesis, it's much simpler to look at a step and say "no, that's
not answering the question correctly".

~~~
louden
This has been my experience as well. To combat this, I always start with a
written analysis plan that forces me to think out all of my analyses.

------
gregunderscorem
I think one of the core problems is summed up down near the end: the idea of
statistics being taught by an apprenticeship model. There's a slightly more
fleshed out version of this idea, also by Dr. Leek, over at
SimplyStatistics[1]

It might be my own statistics-porous media filter, but it seems like this
issue is finally getting the attention it deserves. Hopefully this discussion
will lead to bigger and better things.

[1][http://simplystatistics.org/2015/04/29/data-analysis-
subcult...](http://simplystatistics.org/2015/04/29/data-analysis-subcultures/)

------
T-zex
I have an offtopic question, but believe I could find an answer here. Does
anybody know a statistics equivalent book for Calculus Made Easy by Silvanus
Phillips Thompson [1] Where everything is explained in layman's terms.

[1] [http://www.amazon.co.uk/Calculus-Made-Silvanus-Phillips-
Thom...](http://www.amazon.co.uk/Calculus-Made-Silvanus-Phillips-
Thompson/dp/1456531980)

~~~
pjmorris
'The Cartoon Guide to Statistics', Gonick might be useful. A number of smart
people [1][2] recommend it.

[1]
[http://www.stat.cmu.edu/~cshalizi/36-220/syllabus.pdf](http://www.stat.cmu.edu/~cshalizi/36-220/syllabus.pdf)
[2] [http://mathbabe.org/2014/07/11/the-lede-program-students-
are...](http://mathbabe.org/2014/07/11/the-lede-program-students-are-rocking-
it/)

------
craigching
There was an awesome comment at the end by Stephen Heard:

    
    
      Agreed! Much of the "problem" with P-values comes from
      them being expected to do too much; they aren't (for
      example) indicators of effect size or final proof/disproof
      of an effect's reality. And careful attention to the
      P-value can't overcome problems at the earlier steps you
      lay out here. I would differ in one spot, though: a
      P-value should not be "the last of these [inferential]
      steps" but rather closer to the beginning of critical
      thought! (More about all this here: http://wp.me/p5x2kS-Y.)
    

I like the "p-value should be the beginning of critical though", I'm going to
remember that one.

------
mellavora
Statistics are informative on the data you collected. But if you collected the
wrong data, or collected it in the wrong way (i.e. biased sampling), they are
less helpful at getting to the truth.

Assuming, of course, you want the truth. Lies, damn lies, and statistics as
they say.

~~~
remarkEon
P-value related articles have been popping up on HN recently, but I'd like to
say that another place needing scrutiny is probably back at the beginning in
model specification. Bias introducing model design at the beginning render the
rest of the analysis essentially meaningless.

I know you're joking, but sometimes I wonder if "lies, damn lies, and
statistics" isn't one of the worst phrases ever uttered in the English
language. It's been a way for people to just say "hey, it's just a statistic.
who cares!" and continue doing whatever they're doing. It's become this
terrible meme where someone who openly admits that they "aren't a math person"
gets to thrown down this trump card saying that your analysis is bullshit.

But hey, I'm a baseball fan.

~~~
sesqu
The whole brouhaha over P-values has really been annoying me, specifically
because 95% (Bayesian statistic) of the condemnations I've seen have been
about people picking bad models until they get convenient P-values, and 5%
have been about important misunderstandings about what they measure.

~~~
remarkEon
That is exactly what I'm saying.

------
CWuestefeld
I was just reading a related article in economics. From that article (warning,
while the topic is general, it uses minimum wage research as an example):

It is indisputable that a theory that is inconsistent with empirical data is a
poor theory. No theory should be accepted merely because of the beauty of its
logic or because it leads to conclusions that are ideologically welcome or
politically convenient. Yet it is naive in the extreme to suppose that facts –
especially the facts of the social sciences – speak for themselves. Not only
is it true that sound analysis is unavoidably a judgment-laden mix of rigorous
reasoning (“theory”) with careful observation of the facts; it is also true
that the facts themselves are in large part the product of theorizing.

...

No one who reads this work – and there’s plenty of it – can possibly decide,
based on the data alone, whether minimum-wage legislation does or does not
reduce the employment options of low-skilled workers. It’s not just that the
conclusions drawn from the empirical evidence differ wildly. It’s more that
coming to a conclusion requires reasoning about - theorizing about – the data.
These data (like all data) do not speak for themselves. It’s just not what
data do.

[http://cafehayek.com/2015/04/theorizing-about-the-facts-
ther...](http://cafehayek.com/2015/04/theorizing-about-the-facts-theres-no-
escaping-theory.html)

~~~
baldfat
> politically convenient

I am involved in local politics and currently running for School Board in my
city (We have 14 people running for 5 seats)

Political convenient = how things work locally. If you make a proposal you
have to see it to 4 other people to get it past. If they know their email or
Facebook or Twitter would explode they won't budge. If you show that it is
idea is convenient to them specifically you are a genius.

------
cafebeen
Perhaps the most important point from the article:

"Statistical research largely focuses on mathematical statistics, to the
exclusion of the behaviour and processes involved in data analysis"

If there's a major problem in data collection, cleaning, or hypothesis
generation, it doesn't matter whether NHST or Bayesian methods are used

------
kriro
I'd recommend the following books (in that order) to anyone entering a p-value
laden academic discipline:

"What is a p-value anyway?" -> "Doing Bayesian Data Analysis"

[I found the latter after reading one of the linked "no to p-values" articles
here and I don't want to go back.]

------
hessenwolf
p-values are extremely useful for two things:

1\. Teaching people who specialise in context areas (chemists, engineers) a
pragmatic way to make a decision from data in the presence of randomness.

2\. Automating analyses. (remember your Benjamini-Hochberg)

I'd never actually use it myself. Jeepers, just look at the picture.

~~~
travjones
I agree with #2, but I don't agree with #1. Why would one need a p-value to
make a decision about the significance of a change in a dependent variable?
Confidence intervals provide more information than p-values, and you can use
them to identify significant effects. Moreover, p-values do not indicate the
effect size that was observed. Thus, a very small effect in a very large
sample can have a low/significant p-value.

~~~
hessenwolf
p-value testing and confidence interval testing are not interchangeable?

also, p-values are used to determine the number of effects in a mixed model,
or the number of factors in an ANOVA, etc. Sometimes confidence intervals on
abstract parameters can be a bit less useful.

~~~
travjones
You are correct, they are not interchangeable. I didn't suggest that they
were. I was offering CIs as an alternative for examining effects in general
(e.g., comparing two groups); however, your choice of data representation will
always depend on the question that you're asking. Nonetheless, p-values in and
of themselves provide no information about the magnitude of an effect. Why not
show the actual data?

~~~
hessenwolf
I guess you mean clinical significance.

[http://www.bmj.com/content/348/bmj.g2130](http://www.bmj.com/content/348/bmj.g2130)

This is a secondary issue. For most non-stats people, I wouldn't recommend the
confidence interval as a measure of clinical significance. I'd rather look at
the actual distribution of the two effects and view quantiles of the
predictive interval.

And, yes, in hypthoesis testing, p-values and confidence intervals are
generally interchangeable. All they speak of is the overlap of the sample
distribution of means, really.

------
hnnewguy
Tangentially related, but interesting, EconTalk episode that goes into
p-values related to investment strategies, LHC experiments and other stuff:

 _Campbell Harvey of Duke University talks with EconTalk host Russ Roberts
about his research evaluating various investment and trading strategies and
the challenge of measuring their effectiveness. Topics discussed include skill
vs. luck, self-deception, the measures of statistical significance, skewness
in investment returns, and the potential of big data._

[http://www.econtalk.org/archives/2015/03/campbell_harvey.htm...](http://www.econtalk.org/archives/2015/03/campbell_harvey.html)

~~~
tedsanders
Personally, I thought that episode was rather poor. Harvey made a number of
claims that were just plain false. For example, he said that the p value is
the probability that your result is true (!). He also said that the reason
that physicists at the LHC require 5 sigma is because their data is so noisy
(!).

I think the first one was a miswording, but the second was a serious
conceptual error. I wish Russ had pushed back a little harder on that episode.

------
Osmium
Is there a canonical statistics textbook to recommend to the professional
scientist?

~~~
pjmorris
I don't have a definitive answer, but I'll start the bidding with 'Statistics
for Experimenters' [1]

[1] 'Statistics for Experimenters', G. Box, W.G. Hunter, J.S. Hunter, Wiley

~~~
a_bonobo
I'll raise you Motulsky's Intuitive Biostatistics

It doesn't teach you how to calculate the steps of a t-test, but it will teach
you in which cases to apply what test, the pitfalls of the test, and how to
interpret the results

------
artumi-richard
I stopped with the statistics element of my math degree as soon as I could
because of the disgust I had for the p-value decision thresholds.

~~~
phreeza
Statistics done by mathematicians has very little to do with p-values in my
experience. They are more common when other disciplines try to apply
statistics to their data.

