

Abba - JavaScript a/b testing - olivercameron
http://blog.alexmaccaw.com/abba

======
matsiyatzy
It's a bit confusing that this has the same name as another (pretty well
known) a/b test calculator: <http://www.thumbtack.com/labs/abba/>

~~~
gostevehoward
I'm impressed with how nicely this handles the whole A/B testing stack. But
the statistics have some potentially important shortcomings -- I posted a
github issue about it:

<https://github.com/maccman/abba/issues/3>

(Full disclosure: I authored the ABBA library mentioned by matsiyatzy, so this
is incidentally self-promotional. But I wrote that tool with the hopes of
improving the use and interpretation of A/B test statistics in the wider
community :)

------
danmaz74
Cool. But if you already use google analytics and are lazy like me and don't
want to set anything up server side, you can also consider ABalytics:
<https://github.com/danmaz74/ABalytics>

------
kvirani
The screenshot of the chart shows 217 conversions from 184 visitors, which
isn't right... is it?

~~~
maccman
You're correct - I mocked up that screenshot and looks like I made a mistake,
my bad!

------
karolisd
Very interesting. I'll have to try it out. The first thing I'll look at is how
it deals with the jQuery changes being visible to the visitors (aka
flicker/flashing). Do you hide the page and wait for document to be ready?

Do you do continuous checking of the DOM like Optimizely? Diagram:
[http://support.optimizely.com/customer/portal/articles/45288...](http://support.optimizely.com/customer/portal/articles/452881-how-
when-does-optimizely-evaluate-my-variation-javascript-code-)

~~~
maccman
The trick is to place your a/b testing JavaScript right after the relevant DOM
elements, and not do a/b testing on DOMReady or window.load. That'll ensure
there's no flickering.

~~~
karolisd
I'm just hoping someone comes up with perfect way I can just put a script in
the head and not worry about flickering. So I don't have to hide the body with
CSS and make it visible when the script finishes. Optimizely seems to be
getting there, but it's not perfect. (And kind of awkward, why can't browsers
provide a better way to target elements as they come into existence?)

------
noelwelsh
> If the link is navigating to an external page which you don't control, then
> you have no choice but to cancel the link's default event, wait a few
> milliseconds, then navigate manually:

Why not have a callback on the `complete` method to trigger the navigate when
the request has finished?

 _Update for clarity_ : Presumably the call to the server to register a
conversion uses JSONP or similar. Why not provide a callback for completion
(or timeout) of this call, and use that to trigger navigation away. This seems
more robust and it better encapsulated than requiring the application
developer to setup a timeout.

~~~
ianstormtaylor
You don't want to force your link to depend on receiving a successful response
from the server, because that would be a really bad user experience when it
does break. You should opt to lose your analytics when things break instead.

To do that, you'd need to set a maximum timeout to wait for the response
anyways, so then it's just a (valid) laziness call. If you _absolutely_ need
the best performance then go with a success handler _and_ a maximum wait
timeout. But realistically you want the maximum timeout to be ~300ms anyways,
so why bother with the extra code.

Regardless, it would be nice to be able to pass a callback (whether it's based
on a timeout or receiving a response) right to the `complete` method. Not sure
why you're getting downvoted.

------
togasystems
I wonder if segment.io will be able to interface with this?

~~~
ianstormtaylor
Ian from Segment.io here—randomly happened across this and I was thinking the
same thing while reading it. I'd love to hear about how you were thinking it'd
work, or what the abstraction would look like: ian@segment.io

------
mhmtmhmt
Thanks for open sourcing. How hard is it to convert server-side from ruby to
php ? I was expecting a simple ajax post. However it looks like ruby
everywhere.

------
mrgreenfur
Thank you for releasing this!

------
doctorpangloss
maccman — is not ANOVA the superior comparison?

