

Advanced A/B testing: Make more profit, learn more about customers - efm
https://www.custlabs.com/advanced-ab-testing

======
matthewowen
"data points required for confident results scales linearly with number of
distinct states being tested"

My memory of the maths behind this is poor, but I'm not sure this is actually
true: I would understand that (all things being equal) the number of required
data points grows faster than linearly. The reason is that not only are you
spreading your users across more states, but you also need much stronger
results to make a conclusion: see the use of ANOVA vs (eg) two sample t tests
[http://en.wikipedia.org/wiki/Analysis_of_variance](http://en.wikipedia.org/wiki/Analysis_of_variance).

Of course, if you run a series of base vs variant tests, where the winner
stays on, you run into the exact same class of problem too. Meaningful AB
testing is tricky.

EDIT: when I say not sure, I mean it. I wish I knew this better (rather than
just being aware of the existence of gotchas). If anyone has any great
insights on the impact of this stuff, or how to deal with it, I'd love to hear
them.

~~~
ArikBe
The wording in the original article isn't entirely clear to me. If the test is
across different "states" of one factor, then the number of required
datapoints should be approximately f*n. So for example, testing four different
button placements is like testing four different treatments:

button placement 1 button placement 2 button placement 3 button placement 4

However, if multiple states also coincide with testing multiple factors, such
as four different button placements and also three variations of text on the
buttons. Then the number of combinations is 12. In this case, if we add
another text variation, we would be adding four more treatments.

