> Could you expand on that? Isn't clinical data also observational
Generally in statistics there is a divide between controlled experiments and observational studies. The former is how most medical trials are run and the latter is how most anthropologists work.
In medical trials you can make sure all the demographics, lifestyle differences etc are controlled for before you even start the trial. In the more extreme case of experiments in the physical sciences you can often tightly control for everything involved where the only difference between test and control is precisely the variable of interest.
In anthropology you can't go back in time and say "what if this society had a higher ratio of male/female and didn't go to war?", you can only model what happened and attempt to bake some causal assumptions into your model to see what happened. This is why detailed regression analysis is very important in these fields.
Having a test and control group in your A/B test is not really enough to establish the equivalent of a controlled study of rats in a laboratory, or bacteria in petri dishes. In my experience it really helps to include causal models regarding what you know of your A/B testing population to ensure that what you're observing is really the case. I've had concert examples where it looks like one variant is winning and then checking for the causal assumptions find that we were accidentally measuring something else.
In practice you can reframe all of this into being some sort of ANOVA test and fit it just fine into a classical framework, but I find starting with a Bayesian methodology make the process easier to reason about.
Generally in statistics there is a divide between controlled experiments and observational studies. The former is how most medical trials are run and the latter is how most anthropologists work.
In medical trials you can make sure all the demographics, lifestyle differences etc are controlled for before you even start the trial. In the more extreme case of experiments in the physical sciences you can often tightly control for everything involved where the only difference between test and control is precisely the variable of interest.
In anthropology you can't go back in time and say "what if this society had a higher ratio of male/female and didn't go to war?", you can only model what happened and attempt to bake some causal assumptions into your model to see what happened. This is why detailed regression analysis is very important in these fields.
Having a test and control group in your A/B test is not really enough to establish the equivalent of a controlled study of rats in a laboratory, or bacteria in petri dishes. In my experience it really helps to include causal models regarding what you know of your A/B testing population to ensure that what you're observing is really the case. I've had concert examples where it looks like one variant is winning and then checking for the causal assumptions find that we were accidentally measuring something else.
In practice you can reframe all of this into being some sort of ANOVA test and fit it just fine into a classical framework, but I find starting with a Bayesian methodology make the process easier to reason about.