

Where Are the AB Testing Frameworks? - jbyers
http://www.spiteful.com/2009/01/21/where-are-the-ab-testing-frameworks/

======
patio11
One of my friends, who is regrettably still in stealth mode or I would be
shillin' like a villain, is working on an automated AB testing framework.
Suppose you had a website with a banner, a sidebar, and body text with 3
paragraphs, each with 4 possible variations.

His software, as far as I understand it, genetically engineers the optimal web
page for conversions for you. It apparently is already running on his
(unrelated) software site and he credits it with helping him push through the
$1 million mark. He also says that it doesn't assume there is a static
optimum, but rather that different visitors might need to see different things
(e.g. hypothetically, if someone has seen your screen shot page they probably
don't need more visuals to make the purchase decision, so you'd use more
screen real estate on textual benefits and less on screenshots).

I'm going to be beta testing it on my site, and I'll be blogging progress as
usual, if anyone is interested in hearing about it. I'm pretty excited. I'm
told the tracking code will be ready in about a week or two and the functional
bits ready for beta relatively soon after that. (Nice rolling launch strategy,
incidentally -- gives him time to collect data for the target sites and
resolve the inevitable Analytics Is A Hard Problem (TM) issues, then the black
magic algorithm will have some seed data for when it gets started.)

------
jbyers
Genetify (<http://news.ycombinator.com/item?id=376331>) is one system that was
recently open-sourced. A good discussion about it is on John Resig's site:

<http://ejohn.org/blog/genetic-ab-testing-with-javascript/>

------
dpifke
Would like to think that my comment here inspired this post:
<http://news.ycombinator.com/item?id=433084> :)

As mentioned there, there are a couple of commercial products besides Google
Website Optimizer, one being Optimost (now part of Interwoven). It does a lot
more than just simple AB testing but is quite expensive (we were quoted around
$10k/month when I played with it at my last company).

Genetify looks pretty cool and I'll spend some time playing with it. One thing
I wish all these frameworks had is hooks for testing things server-side rather
than with client Javascript. This would make it a lot easier to test process
flows (i.e. "one-page form versus wizard") in addition to individual page
elements.

If there are other open-source options out there, I'm looking forward to
hearing about them.

------
nonrecursive
Here's an ab testing plugin I wrote for rails:
<http://github.com/flyingma/rails_ab_split_tester/tree/master>

I'm hoping going to get something similar working for merb soon.

------
aneesh
<http://exp-platform.com/> is an A/B testing platform used internally at
Microsoft that will be made available externally at some point.

------
burnout1540
The author mentions Google Analytics, but it's not entirely clear whether he
means that to include Google Website Optimizer
(www.google.com/websiteoptimizer). I just completed a project with it and it
is quite well implemented. The biggest downside is that you have to rely on
JavaScript and requests to third-party servers.

~~~
slackerIII
Ah, good catch. I thought those were rolled up together, but I've edited the
post to make it clear I'm talking about GWO.

------
trapper
I built one years ago (2001ish) but never even tried to commercialise it as
there are tons of patents on it, with relatively big players. Perhaps they all
folded, I haven't kept up with it.

~~~
slackerIII
That's an interesting idea. Even if the companies went away, I'm sure the
patents didn't. Boo...

~~~
trapper
I am sure that's why google didn't implement it earlier; perhaps they have
some licensing deal with those companies now if they are still around. It's
pretty simple technology.

------
wmeredith
Lyris HQ's Clicktracks Analytics solution has something in place, but it's
hard to implement.

------
terpua
It would be cool if Weebly had this as a feature.

