

How to A/B Test Using Javascript in Your Posterous Blog - rj_metrics
http://themetricsystem.rjmetrics.com/how-to-ab-test-using-javascript-in-your-poste

======
3pt14159
I say this all the time, but be very very careful split testing in JavaScript.
It is best left for very small elements on your site. Otherwise your results
will be misleading. Once you get to a nontrivial size implement it server
side.

~~~
sshrin
Can you elaborate on why you think the results will be misleading? Does it
have to do with data integrity issues of some sort?

~~~
3pt14159
Because for large elements there is a delay, especially when your user has a
slow browser and is talking to a third party service that is far away.

For example, say you ran a company that had about 100 paying users, almost all
of which were in the SV tech scene. A fine opportunity to use JS because they
all have fast browsers and the odds that the delay between KISS Metrics or
Mixpanel and them are fairly low.

If, on the other hand, you sell Bingo Card Creators to thousands of teachers
around the globe the latency is going to kill you. A teacher in England will
see the button (s)he wants to click on, then all of a sudden it is gone and
replaced with a intro video, or, even worse, the prices have all gone up and
there are different options on the pricing page.

Not only is your split test reporting the wrong winner (because of a negative
bias on the B side of the test) but it can also erode the confidence visitors
had in your service.

