

A Roadmap To Becoming An A/B Testing Expert - ra00l
http://www.smashingmagazine.com/2014/07/11/roadmap-to-becoming-an-a-b-testing-expert/

======
splitforce
Thanks for the article Kevin, but I am afraid that most of your audience will
find that just tweaking colors or copy will not produce any meaningful results
for their businesses. Let me explain…

We’ve found that a successful approach to A/B testing is really dependent on
the type of company you’re operating and product that you’re offering. Small
cosmetic changes to the UI or copy often result in equally small changes to
click-thru or conversion rates, and so these A/B tests require relatively
greater levels of statistical power in order to achieve significance.

For mega-traffic companies like Google or Amazon, these kinds of tests are
worth the cost of testing because a sub-1% lift still contributes
substantially to their bottom line.

But for everyone else, ‘shallow’ A/B tests of a button color or call to action
will often yield inconclusive results. Here’s an article from the founder of
GrooveHQ detailing such an experience: [http://www.groovehq.com/blog/failed-
ab-tests](http://www.groovehq.com/blog/failed-ab-tests).

If you’re running a small or medium business – or even a larger one that does
not have the scalable testing practices of a tech giant like Amazon in place –
testing deeper changes to the product, UI layouts or entire UX workflows are
what move the needle. This is what we’re now calling ‘empathic A/B testing’ –
where tests are designed with empathy for users.

If you ask the questions: What changes can I make to my product or website
that would motivate my users to take the actions I want them to take? What are
they looking for? What do they care about? And why? More often than not, I
think you’ll find that the answer is not ‘a different button color’

In the end, A/B testing is really a very unsophisticated way of answering the
question ‘What works better?’ because you are sending a fixed proportion of
your users to a suboptimal variant for the duration of the test. We’ve done a
lot of research into better solutions to this problem, and have found that a
dynamic approach using a learning algorithm always leads to faster results and
higher average conversion rates. You can read more about that here:
[http://splitforce.com/resources/auto-
optimization/](http://splitforce.com/resources/auto-optimization/)

