That is a very reasonable question, and in this context we might reasonably say that "this [individual] outcome is due to chance" means the same thing as "the null hypothesis we stated in our introduction is platonically correct".
But I don't really see the relevance to this discussion?
Suppose you nail down a null hypothesis, define a similarity metric for data, run an experiment, and get some data. The p-value you calculate theoretically tells you this:
If the above-mentioned hypothesis is true, then X% of all data looks like your data
It doesn't tell you this:
If you have data that looks like your data, then there is an X% chance that the above-mentioned hypothesis is true
Those are two unrelated claims; one is not informative -- at all -- as to the other. The direction of implication is reversed between them.
Imagine that you're considering three hypotheses. You collect your data and make this calculation:
1. Hypothesis A says that data looks like what I collected 20% of the time.
2. Hypothesis B says that data looks like what I collected 45% of the time.
3. Hypothesis C says that data looks like what I collected 100% of the time.
Based only on this information, what are the odds that hypothesis A is correct? What are the odds that hypothesis C is correct? What are the odds that none of the three is correct?
This is getting deep into the weeds of the philosophy of science. It is crucially important to choose good hypotheses to test. For example:
> Hypothesis C says that data looks like what I collected 100% of the time.
What this tells you depends entirely on what hypothesis C actually is. For example, if C is "There is an invisible pink unicorn in the room, but everyone will deny seeing it because it's invisible" then you learn nothing by observing that everyone denies seeing the unicorn despite the fact that this is exactly what the theory predicts.
On the other hand, if C is a tweak to the Standard Model or GR that explains observations currently attributed to dark matter, that would be a very different situation.
> It is crucially important to choose good hypotheses to test.
But if you were able to do that, you wouldn't need to test the hypotheses. You'd already know they were good.
I'm intrigued as to why you picked those two examples. They differ in aesthetics without differing in implications, but you seem to want to highlight them as being different in an important way!
Seriously? You don't see any substantive difference between an explanation of dark matter and positing invisible pink unicorns? How do I even begin to respond to that?
Well, let's start with the obvious: there is actual evidence for the existence of dark matter -- that's the entire reason that dark matter is discussed at all. There is no evidence for the existence of invisible pink unicorns. Not only is there no evidence for IPU's, the IPU hypothesis is specifically designed so that there cannot possibly be any. The IPU hypothesis is unfalsifiable by design. That's the whole point.
If the invisible pink unicorn hypothesis was true, what about the world would be different?
If the MOND hypothesis was true, what about the world would be different?
The whole reason we have a constant supply of theories attempting to explain observations currently attributed to dark matter in terms other than "dark matter" is that people feel the dark matter theory is stupid. There's nothing else to it. I assume you feel the same way about unicorns. What's the difference supposed to be?
> There is no evidence for the existence of invisible pink unicorns.
You need to be careful here too. The fact that a theory is false does not mean there is no evidence for that theory.
> The whole reason we have a constant supply of theories attempting to explain observations currently attributed to dark matter in terms other than "dark matter" is that people feel the dark matter theory is stupid.
No, that's not true. The reason we have a "constant supply" of dark matter theories is that all of the extant theories have been falsified by observations, including MOND. If this were not the case, dark matter would be a solved problem and would no longer be in the news.
> The fact that a theory is false does not mean there is no evidence for that theory.
What makes you think the IPU theory is false? The whole point of the IPU hypothesis is that it is unfalsifiable.
But I don't really see the relevance to this discussion?
Suppose you nail down a null hypothesis, define a similarity metric for data, run an experiment, and get some data. The p-value you calculate theoretically tells you this:
If the above-mentioned hypothesis is true, then X% of all data looks like your data
It doesn't tell you this:
If you have data that looks like your data, then there is an X% chance that the above-mentioned hypothesis is true
Those are two unrelated claims; one is not informative -- at all -- as to the other. The direction of implication is reversed between them.
Imagine that you're considering three hypotheses. You collect your data and make this calculation:
1. Hypothesis A says that data looks like what I collected 20% of the time.
2. Hypothesis B says that data looks like what I collected 45% of the time.
3. Hypothesis C says that data looks like what I collected 100% of the time.
Based only on this information, what are the odds that hypothesis A is correct? What are the odds that hypothesis C is correct? What are the odds that none of the three is correct?