

The Disadvantages of A/B Testing - fezzl
http://blog.zuupy.com/the-disadvantages-of-ab-testing

======
blackRust
"1. A/B testing requires sacrificing some conversions as part of the
experiment." Yes, because it is a net gain if you do it right. If you want 5
signups today instead of 50 in two weeks you're doing it wrong.

"2. A/B testing measures only quantity, not quality." If you have anything
that can quantify your 'quality' of customers then include it in your metrics
(I understand this can be hard). If you don't then it is not quantifiable and
any analysis of 'quality' won't be based on statistics.

I agree on the other points. Overall I think A/B & multivariate testing is a
good approach but can significantly be improved.

It seems like we use it in 'one size chunks' rather than as continuous
improvement. Yes we do it ever 'x' days/weeks. What would be interesting is
some kind of machine learning algorithm applied to A/B testing. It has the
success metrics, you set a rotation period and it focuses in on the part that
works.

You would have to integrate a factor for #5 "Things change" so that the
algorithm has the flexibility to adjust to those changes.

