

The Math Problem That A/B Tests Should Be Trying To Solve - ewjordan
http://en.wikipedia.org/wiki/Multi-armed_bandit

======
ewjordan
Just to be clear: I'm not necessarily suggesting that full multivariate A/B
testing is the best thing to shoot for, I realize very well that especially in
site design, different splits cannot be composed together.

However, related to a discussion this past week where it became clear that
periodically checking confidence intervals until they "hit" was a terrible way
to test because of the high likelihood of picking the wrong result, I _am_
suggesting that the two-armed bandit problem should be considered mandatory
knowledge for anyone working on A/B testing frameworks. That problem is
_literally_ what A/B testing tries to solve (try to make choices between two
options to maximize payoff when you don't know the payoffs of either choice
beforehand, taking into account the fact that trials spent building up
certainty by testing the payoffs are sometimes wasted on the least valuable
option).

I'd have linked to the two-armed bandit problem if there was a page on that,
but it just redirects...

