

6 A/B Tests That Did Absolutely Nothing for Us - davidw
http://groovehq.com/blog/failed-ab-tests

======
splitforce
Nice article as always, thanks Alex!

What I’ve found is that a successful approach to A/B testing is really
dependent on the type of company you’re operating and product that you’re
offering. Small cosmetic changes to the UI or copy often result in equally
small changes to click-thru or conversion rates, and so these A/B tests
require relatively greater levels of statistical power in order to achieve
significance.

For mega-traffic companies like Google or Amazon, these kinds of tests are
worth it because a sub-1% lift still contributes substantially to their bottom
line. They also have the traffic numbers to properly power tests of smaller
changes in a reasonable amount of time.

But for everyone else, ‘shallow’ A/B tests of a button color or call to action
will often yield inconclusive results because they don’t the traffic numbers.
For these types of companies, we’ve seen that deeper changes to the product,
UI layouts or entire UX workflows are what move the needle. Designing these
tests require more thought and development work up-front – but at least you’ll
be making substantial improvements in an experimentally rigorous way instead
of just spinning your wheels with some one-off design tweaks.

To avoid these kinds of disappointing tests, another thing to consider is
setting a minimum detectable effect. The idea here is that validating a small
change in improvement requires more statistical power (i.e.: more test
subjects) than validating a large change, and at some point in order to
justify a continuation of the test you’ll want to achieve some minimum amount
of lift. Once you can say with statistical confidence that this desired lift
isn’t achievable, you can stop the test earlier and move on to the next.

Most importantly, you should be designing these tests with empathy for your
audience. Ask the questions: What changes can I make to my product or website
that would motivate my users to take the actions I want to them to take? What
are they looking for? What do they care about? And why? More often than not, I
think you’ll find that the answer is not ‘a different button color’ :-D

In the end, A/B testing is really a very unsophisticated way of answering the
question ‘What works better?’ We’ve done a lot of research into better
solutions to this problem, and have found that an automated approach using a
learning algorithm almost always leads to faster results and higher average
conversion rates. You can read more about that here:
[http://splitforce.com/resources/auto-
optimization/](http://splitforce.com/resources/auto-optimization/)

