> I understand that in the strictest bayesian sense, any data that doesn't could directly reject a hypothesis, but doesn't, 'partially supports' the hypothesis.
>
> But, I'm not sure that, in practice, its meaningful to differentiate between which of the two methodological problems is the generally bigger or smaller issue.
I'd disagree there, you can do more than simply throw up your hands and say correlation!=causation. Some of the work stemming from Pearl's causality formalism has been about conditions under which one can make causal inference even in the absence of an explicit randomization step, and one can attack it from another direction by compiling correlations which have been later examined with randomization, and estimating how many of the correlations turned out to be causation in the same direction (the numbers tend to look like ~10%) and how much they are due to other factors.
> Isn't [potential selection bias etc.] just as bad as [potential expectancy effects etc.]
No. Expectancy effects can be manipulated and measured and one could arguably then adjust for it in other results where expectancy is mixed in. It's a fact about human psychologies, as measurable as anything else there. Selection effects are too wild and unpredictable to hope to do such a thing.
I'd disagree there, you can do more than simply throw up your hands and say correlation!=causation. Some of the work stemming from Pearl's causality formalism has been about conditions under which one can make causal inference even in the absence of an explicit randomization step, and one can attack it from another direction by compiling correlations which have been later examined with randomization, and estimating how many of the correlations turned out to be causation in the same direction (the numbers tend to look like ~10%) and how much they are due to other factors.
> Isn't [potential selection bias etc.] just as bad as [potential expectancy effects etc.]
No. Expectancy effects can be manipulated and measured and one could arguably then adjust for it in other results where expectancy is mixed in. It's a fact about human psychologies, as measurable as anything else there. Selection effects are too wild and unpredictable to hope to do such a thing.