I can see the analogy here but a test like that being binary throws things off and that's also super asymmetrical error. Weren't we sampling a gaussian?
It’s the same principle. The “trivial interval” from the first link (the one you called pedantry), which is applicable to a Gaussian and tweakable to have different coverage than 50%, clearly shows that being an n% confidence interval, on its own, doesn’t guarantee the ability to directly translate that n% into post-data inferences. Likewise, in the truncated exponential example, when you compute that confidence interval of [12.1471, 13.8264], you know that the probability that it contains θ is 0%. In the Cauchy example, when you compute [-2.31, 10.31] as your 90% CI from the two samples 3 and 5, you can then compute that it actually has a >99% chance of containing θ (table I).
The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.
The examples that actually ruin the number so far have been asymmetrical. Is there a way to do it with a gaussian? Especially if you're trying to make a reasonable internal?
> The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.
Sorry, I don’t follow. Let’s say I want to compute a 50% confidence interval for the unknown mean of a Gaussian distribution. I sample two numbers from the distribution, get 9 and 7, compute the interval according to the trivial procedure and get (-∞, ∞). Does the interval (-∞, ∞) have a 50% probability of containing the mean of that Gaussian distribution? I would think it’s closer to 100%.
If what you are saying is “it’s meaningless to talk about the probability of that specific interval containing the unknown-but-fixed parameter” then that’s the purely frequentist view and then you also agree that it’s meaningless to say that [-2.31, 10.31] has a 90% chance of containing the location parameter of the Cauchy distribution that happened to yield the samples 3 and 5. Incidentally, what asymmetry are you referring to in the Cauchy example?
I misread the Cauchy example since I was going too fast, nevermind the symmetry part.
> then that’s the purely frequentist view and then you also agree that it’s meaningless
I'm saying that when you hit "meaningless" you can back up a step to where you actually had randomness and look at that distribution, which gets rid of a lot of these issues.
But after looking at these examples I think it only makes sense in limited circumstances to do that. Like in the trivial example: your final distribution isn't based on the probability of the mean being any particular number. The only probability was back a step and that was 50%.
At this point I still don't think it's objectively wrong to say a particular interval above is 95% likely, but there's too many ways to interpret the statement so nobody should say it is.
The way we're calculating that these intervals are "wrong" is by looking at all the possible parameters that could have given us the samples we got, and checking how often the range contains the parameter. That's a useful calculation but is it the one people expect? I think that depends on the situation. Treating the parameter as being the thing we sample over is misleading, but treating it as fixed is also misleading.
