Many A/B tests as RCTs do suffer from the fact that they're randomizations of convenience sample of users coming to a web site, but it's always seemed to me that randomization addresses most confounders due to population heterogeneity. Is there some specific confounding variables you worry about that are not addressed by randomization in this way?
One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back.
In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?
One challenge for me is how to explain to marketers what the "Probability A is better than B" means from a Bayesian perspective. How do you convey to them that such a probability is not a frequency of an event? I haven't met anyone who is not a statistician that won't default to "you know, how often it happens" when pressed to define probability. My attempts to engage marketers on this fall like water off a duck's back.
In general I prefer NHST methods because I often require reasonably tight error control. Do you do any kind of calibration to ensure error rate control when doing your procedures? Or are your clients comfortable with less tight inferential error controls?