

Effective Web Experimentation as a Homo Narrans - mcfunley
http://mcfunley.com/effective-web-experimentation-as-a-homo-narrans

======
noelwelsh
Great post.

One of the advantages of the bandit algorithms we favour is that you have to
be explicit about your quality metric up-front. This can also be a
disadvantage as it can be difficult to tradeoff different metrics, but on
balance we find it helpful for our clients.

I think one of the main issues with A/B testing, which was also present in the
article, is the over-reliance on p-values without considering effect size. The
changes show were significant, but none of them had a large effect. The
largest was a 9% increase ("Registered") but here we're talking about going
from 0.46% to 0.42%, which is pretty much nothing to nothing.

~~~
mcfunley
Yes, that's a great point. Although that test runs on our search pages, so
0.46% of that traffic is about 4,000 registrations. And the difference between
0.46% and 0.42% is about 300 registrations.

