

A/Bingo: OSS Ruby on Rails A/B Testing.  No Javascript.  Two lines of code. - patio11
http://www.bingocardcreator.com/abingo/

======
patio11
A quick aside: A/Bingo exists primarily because of HN, whose articles and
discussion drove me onward to finishing it. Thanks!

If you have questions, I've got answers, at least for the next hour or so.
(Tonight is date night. I'm dedicated to my work, but I'm not _that_
dedicated.)

~~~
imajes
kinda wish this was on github. :)

------
charliepark
This looks like a terrific resource. I was just setting up an A/B test the
other day with Google Website Optimizer ... I just wanted to test a single
word change. To go through GWO, I had to set up a multivariate test, with
multiple js snippets going in multiple places on multiple pages.

I haven't installed A/Bingo on any live sites, but I'm looking forward to
kicking the tires. Thanks for releasing it.

Since it's open source, have you thought about setting it up at Github? Since
that's becoming such a standard resource, it'd make looking through it easier.

~~~
patio11
[Edited to add:

 _To go through GWO, I had to set up a multivariate test, with multiple js
snippets going in multiple places on multiple pages._ That was EXACTLY my pain
point with their multivariate option. It took me 15 minutes to set up a single
test on a single call to action, and then 15 minutes more to tear it down when
I was done.

Testing a call to action should look like:

<%= ab_test("call_to_action", ["Buy it today!", "Get it today!"]) %>

And now, it does.

]

 _Since it's open source, have you thought about setting it up at Github?_

Yes, I did think about it. We're all businessmen here, right, so we can
discuss the following in a mature fashion:

1) Github gives out free hosting to OSS projects in the hopes of encouraging
people to host OSS projects of value on Github. This means Github collects
links and mindshare, which they then monetize through selling hosting
services.

2) I write OSS (and provide statistics, and write articles, etc) in the hopes
that it helps other businesses. This means I collect links and mindshare,
which I then monetize through selling software to elementary schoolteachers.

Given that I did the work, I think it is reasonable that I collect the
majority of the benefits, rather than giving them away to a hosting provider.

If copy/pasting the git clone line is too much work to see the code, you may
wish to reconsider using A/Bingo. I made it easy -- I can't make it THAT easy.
But if you really want to see it in Github, it is MIT licensed, upload it to
your own account and then you can view it in your browser to your heart's
content.

~~~
markpercival
I would disagree on one point, Github isn't just for hosting, it's for
collaboration. That's the reason most people use it for OSS project.

I can, and do, host my own git repos on a private server, but when I open
source something I like having it on Github where people can fork it and send
me pull request for new features and bug fixes.

------
scottru
I'm a very active GWO user. I haven't tested A/Bingo yet, but based on the
description, there's one other advantage you have: I don't have to rewrite erb
as javascript in order to execute various tests.

If a lot of your view layer is written in erb (likely b/c it's not javascript-
heavy), then to use GWO, you have to basically rewrite all of it in js, since
GWO dynamically inserts client-side code into the spots you specify.

So if you have a display loop that uses a bunch of object properties in your
erb, for example, you have to copy those objects/values to js variables and
move the loop to js. It's all doable, it's just very non-DRY and just
altogether unpleasant.

(The other option with GWO is to use the Javascript to load partials, which I
do now, but which has some unusual properties, like lazy loading.)

Having the a/b decision be server-side is a nice thing. I'm looking forward to
trying it out.

(Also, not having to pull out piles of GWO code every time is a nice thing
too.)

------
defen
This is great, thank you. I was literally going to start working on a Rails
A/B framework after reading HN today. Funny how that works.

Edit: In the newest version of Rails (2.3.3) and IIRC a few versions before
that as well, the timestamp field is called _created_at_ , not _created_on_ \-
I was getting an error when running the migration.

~~~
patio11
Thanks for the bug report. I can't address it right now as I'm at an Internet
cafe, with my laptop spending the night at my designer's house, but I'll fix
it in the morning. Off the top of my head I don't remember anything that
actually uses the timestamps, so feel free just to comment out the line (was
it assigning an index, maybe?) that caused the exception.

~~~
charliepark
You're right; it was assigning an index, on line 11 of the migration. Changing
it to "created_at" works fine.

~~~
patio11
And fixed. Thanks for the report.

------
callmeed
This is awesome Patrick, thanks.

------
trapper
What's the minimum effect size you consider worthwhile?

~~~
patio11
Do you mean "How positive does it have to be for you to not revert it when you
end the test?"

I'd collect 2% improvements all day long if they looked statistically
significant. That being said, my mental goal is to run one test a week, and to
collect 5% improvements on half of my tests. The miracle of compound interest
means that, if I sustained that for a year, I'd increase sales by 250%. You
can see my stats and tell that I've never had a year over year improvement
that good, but now since the cost of A/B testing just got muuuuuuuch cheaper
for me, I'm optimistic.

~~~
trapper
Yes. Effect size is the difference in means in proportion to the pooled
standard deviation to the two treatments, and is a great measure of how large
the difference is. All groups are different given large enough n. Whether that
difference means anything is dependent on the effect size.

When the effect size is large you know that whatever effect you are seeing is
much less likely to be chance. It's also less susceptible to large differences
in variation, which the tests you are using to test significance use as an
assumption (variation is equal).

