

The Monty Hall problem and what it has to do with A/B testing - paraschopra
http://visualwebsiteoptimizer.com/split-testing-blog/the-monty-hall-problem/

======
klochner

        Now, what is the comparison between [The Monty Hall] problem and the
        sample size blog?
    
        Well, both are about the usage of additional information
        to optimize decision making.
    

Am I the only one feeling disappointed with the payoff here? The answer ends
up being "almost nothing".

It's the geek equivalent of TV gossip programs advertising "Find out next what
crazy thing Tom Cruise said to Tiger woods . . . " with the result being "They
actually didn't talk."

~~~
keesschippers
Kevin,

Bare in mind this article was written as a follow up on another article that
amply describes the concept of statistical power.

I think a comparison between the two is justified. Both are about the use of
additional information to enhance decision making and both are obviously
difficult to grasp.

There is a conceptual resemblance between the problem many scientists had a
long time ago understanding the dynamics of the probabilities involved in the
Monty Hall Problem and the problem many marketing data analists have nowadays
trying to understand the dynamics of statistical power when designing
experiments.

The practical resemblance between the two is that failing to apply the logic
behind the Monty Hall Problem will make you throw away half of what you could
have had. Not applying the logic of statistical power makes many data analysts
and thus organizations throw away half the knowlegde they could have had.

Apart from the resemblance between the two puzzles, the practical consequences
of such misconceptions are quite worrying to say the least. It’s for that
reason I make these efforts to explain the matter.

Regards, Kees

------
jules
All of this madness is elegantly tackled by Bayesian statistics. There are
usually two roadblocks to using a Bayesian approach:

1\. It's computationally expensive.

2\. You need a prior.

#1 is not an issue at all here since the problem is tiny.

Companies like Visual Website Optimizer are in an excellent spot for #2:
because they have data from many customers, they can find a very good prior.

------
jedberg
In summary: Make sure your sample size is large enough so that it is possible
to get a statistically significant result.

~~~
keesschippers
Indeed, or, if that is not possible, consider lowering reliability. Business
situations are different from academic settings where an alpha of 5% is more
or less the rule. We do not look for publications in scientific journals, nor
do we engage for instance in medical research – and even there -, we are in a
business situation trying to optimize decision making.

Up to us to decide for an alpha of 10% or 20%, as long as we are fully aware
of the consequences of such choices. I am not advocating ‘going easy’ on
statistical significance, I am in favour of weighing consequences and such can
only be properly done if one masters the dynamics of reliabilty and power of a
test. That could also mean you wanted a reliability of 99% and a power of 99%,
it all depends on the risks you want take with your specific research
questions.

