

Poor Man's Template A/B Testing (in Django) - amcgregor
http://tech.matchfwd.com/poor-mans-template-ab-testing/
Ever wanted to A/B test your templates in Django and didn't know where to start? Here is a 50 line function as a drop-in replacement for render_to_response with full breakdown of how and why.
======
DrDub
Everytime I read about A/B testing I remember this posting about why you do
better than A/B using solutions to the multi-armed bandit problem:
<http://stevehanov.ca/blog/index.php?id=132>

~~~
amcgregor
Convienently, the solution I wrote allows you to utilize two or as many
templates as you want, add and remove to the directory as you wish. For us,
setting off a handful of template options for two weeks then seeing which one
has higher conversion is extremely simple (no need for 'reward' function as
such) and effective (it's a fair distribution across the options). We
explicitly wanted to avoid needing to track the number of times each
possibility is viewed.

I think "better" needs to be defined. "More accurate results?" Possibly, but
probably not by much. Less disturbance to general conversion rates (since the
'successful' case is presented most often barring random variability), sure.
But the point is to try out every option, not try out the seemingly more
successful option the majority of the time.

