

ABalytics.js: Client-side A/B Testing With Google Analytics - danmaz74
http://danmaz74.me/2012/10/10/abalytics-js-client-side-ab-testing-with-google-analytics/

======
yahelc
You can't reliably use Array#concat on the _gaq object. Once Google Analytics
loads, it replaces the _gaq array with a custom object that has it's own
push() method. That object doesn't have concat() on it.

So, your core coding style will only work with very particular GA
implementations; anything that deviates from that could very easily break or
be subject to race conditions.

You'll also get better results if you use events rather than custom variables;
custom variable reporting isn't as flexible and can be very confusing (ie,
visit start). It'll also mitigate the need to use concat() to cut the line,
since Events can be fired at any time (whereas custom variables need to happen
before the first hit of the session to be well-reported).

I also really like the idea, though. I actually pass all my Optimizely test
results as Google Analytics events with this snippet:

    
    
         if (window.optimmizely && optimizely.variationNamesMap) {
    		var optmap = optimizely.variationNamesMap;
    		for (var expt in optmap) {
    			if (optmap.hasOwnProperty(expt)) {
    				_gaq.push(["_trackEvent", "Optimizely", optimizely.allExperiments[expt].name, optmap[expt], 0, true]);
    			}
    		}
    	}

~~~
danmaz74
Concat is only called before GA is loaded, so I don't think any race condition
could happen.

~~~
yahelc
Only if everyone loads it the same way. But there are lots of different ways
to load GA; the only reason _gaq.push() isn't scary and doesn't raise the
specter of race conditions is because of the 'spoofing' GA does.

There are LOTS of different versions of the GA snippet floating around the
internet; the assumptions this configuration makes would only work for some of
them.

For example, some people load GA like this:

    
    
         <script async src="//google-analytics.com/ga.js">     </script>
         <script>
         var _gaq = _gaq || [];
         ...
         </script>
    

Folks who do this will break at minimum with older browsers that don't support
async; not sure what it would do with async support, though.

It seems unnecessarily risky.

~~~
danmaz74
Ok, now I understand what the issue is. I guess a good solution would be to
pass _gaq to my library initializer and push every value with a for cycle,
instead of using concat. Thanks for the tip.

------
Timmy_C
Checking out the source code, this isn't going to work in IE8 and earlier
without some small tweaks. IE didn't support
`document.getElementsByClassName(string)` until IE9. It would be a small
rewrite to make it work with IDs instead of classes (or you could use a shim).

[EDIT]: . . . But I like the idea. Might even use it next time I'm working
with GA.

~~~
danmaz74
Thanks for the warning! I initially developed this with jQuery, and only after
seeing how little of it was needed decided to get rid of it - but I forgot to
make all the due diligence. Later or this weekend I'll fix this - but if in
the meantime anybody wants to send a fix or a pull request, I'll be glad to
use it.

------
btilly
My first thought upon seeing this was, "Where are the statistics so that you
know if you are just ahead by chance?"

My second thought was, "Testing is so much better than not testing, this is
still worthwhile."

Seriously, if you aren't testing, do it. If you are, this is something to
graduate out of.

~~~
yahelc
I made a bookmarklet for easily calculating statistical significance,
specifically for this use case.

After enabling the bookmarklet, you click the numbers you need and then it
sends it to an A/B testing calculator hosted by the folks who made ABBA

Project: <https://github.com/yahelc/ABBA-bookmarklet> Sends to here:
<http://www.thumbtack.com/labs/abba/>

------
ohashi
Great job. I came up with my own ghetto solution using event tracking, yours
is definitely better.

------
ckluis
Very Nice! Personally, I would have loved a code example inside the post.

Thanks for going MIT.

~~~
pella
more information about usage:

<https://github.com/danmaz74/ABalytics>

~~~
ckluis
Oh, I opened that. I just meant my favorite - check out my cool tool posts
include samples of cool tool being cool. :)

------
theseanstewart
Just me, or does anyone else miss Multivariate testing? It's impossible for me
to use A/B testing with shopping cart software when it loads dynamic content
(like the cart or checkout processes). Unless I'm doing it wrong?

~~~
siddharthdeswal
Content that loads dynamically won't force you to use Multivariate Testing. In
fact, most commercial services implement both A/B and Multivariate tests using
client side JS.

Also, you can A/B test dynamically loading content so I think you just need to
find out how to do it using your chosen software.

------
jentulman
looks nice and this is just an initial thought, it looks like the holding
content in the HTML is replaced by either case A or B from the script, will it
run with the holding text being case A and swapping out only when B is picked
by the script therefore avoiding a possible content flicker some of the time?
of course this may not be desirable on larger content.

~~~
danmaz74
Yes, it would work - but I wouldn't advice doing so:

    
    
       * If the flicker isn't noticeable, it doesn't matter either way
       * If the flicker is noticeable and disturbing, having it only for one version could skew the results in favor of the non-flickering version

