

P-values are inconsistent - ekiru
http://www.johndcook.com/blog/2010/03/03/p-values-are-inconsistent/

======
roundsquare
How do you do a statistical test against "mu is between a and b?"

If H0 (null hypothesis) is mu = a (and you know sigma = 1 and its normal) then
you can pretty easily calculate the p value from an observation x by taking z
= x - u and checking that against the normal distribution.

However, how do you do that is H0 is a < mu < b? Do you assume that mu is
uniformly distributed between a and b? If so, then H' does not imply H. E.g.
if H is true then pr(mu > 0) is 0.5, but if H' is true then pr(mu > 0) =
0.52/1.34 < 0.5.

Or, am I wrong about how to do the hypothesis test?

~~~
sesqu
You use the cumulative distribution function, which is defined as P(X≤x). More
specifically, denoting the cdf of the standard normal distribution by 𝛷,
P(X̅-a𝜎<µ≤X̅+b𝜎) = P(a<Z≤b) = P(Z≤b)-P(Z≤a) = 𝛷(b)-𝛷(a) for large samples.

Edit: Err, maybe I got that wrong. Doing the above calculation would favor the
broader hypothesis (p=0.047 vs. p=0.043).

~~~
roundsquare
Well, if we leave the details aside, you do _something_ where you take a
probability distribution between a and b right? If thats the case, I think my
original point stands, neither hypothesis really implied the other one because
they say different things about the probability that x is in certain ranges.

I'm sure there are other problems with using the p value as a measure of
certainty (though I don't know them) but this specific criticism seems silly.

~~~
sesqu
I absolutely agree. I can't see anything wrong with my calculation above,
which leads me to conclude the blog post or the article it is based on
misunderstands – or at the very least miscommunicates – the problem, if indeed
there is one.

The actual problem might be that standard statistical tests are difficult to
use, but then isn't that why the theory had to be developed in the first
place?

