
Ask HN: What is your one tip about A/B testing you would share? - raycloyd
Wondering what everyone would tell someone new to a&#x2F;b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!
======
Someone1234
That most A/B tests don't prove what they aim to prove. People just add a new
version of the site, then wait until 100 people have seen it, measure a small
improvement and migrate to the new version.

However if you then run the test a second time, but make the "old" site the
alternative, you might find it too sees a small measure of improvement and
thus you could bounce back and forth between A and B until the end of time.

I am not a statistician so I won't try to give you advice on what a
statistically significant result is. However many /many/ articles have been
written on that topic, and many products have been designed which you can slot
your data into to see if the 'B' option is legitimately better.

------
onion2k
Learn what "statistical power"
([http://en.wikipedia.org/wiki/Statistical_power](http://en.wikipedia.org/wiki/Statistical_power))
means, and then understand that the fact n% of users prefer one option _doesn
't_ necessarily make it a good idea. Statistical significance is _far_ more
important than the actual result. This is a really good paper about it:
[http://www.qubitproducts.com/sites/default/files/pdf/most_wi...](http://www.qubitproducts.com/sites/default/files/pdf/most_winning_ab_test_results_are_illusory.pdf)

------
dairgram
Before you get too far, try an A/A test. By this, I mean let the A and B
choices be identical. You would certainly expect the outcomes to be equal.
Right?

I have seen statistically significant differences in outcomes in A/A testing.

A/B testing has value but being sure to A/A test may temper your expectations
and/or point at problems in your setup before you get too far.

------
domrdy
[http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky)
\- to make clients 'stick' to upstreams, if you're using nginx.

------
mtmail
If you radically change a feature then regular users will first start playing
with it. Simply because it's new. In that case your test needs be longer or
exclude regular users.

------
seekingcharlie
Ensure that your testing things that are actually going to impact your
conversion rate or goals. You don't have to test everything - just the things
that matter.

------
siddharthdeswal
Don't A/B test your credibility. Not everything should be (or can be)
subjected to hypothesis testing.

------
hkiely
Choose and test one independent variable at a time. Then, check your results
for statistical significance.

------
catman01
Kittens will always improve conversion rates.

