

Show HN: A/Bingo for PHP - sdrinf
http://github.com/waverz/cog/blob/master/cog-guestbook/engine/cog_splittest.php

======
patio11
Thanks -- PHP developers need better alternatives for A/B testing than Google
Website Optimizer. Your code is about the minimum viable A/B testing library
for a developer: you can use it to start getting your feet wet with A/B
testing done the right way.

I think if you start using it in production you will quickly understand that
some of the implementation choices you've made are going to cause you pain,
but that is less important than doing A/B testing. You can always rewrite the
internals later. (I've done it a few times.)

~~~
sdrinf
Thanks.

Would you mind pointing out implementation choices, that will come back biting
us down the road? :)

~~~
patio11
Your mileage may vary, and I don't guarantee I'm reading your code right, but:

1) You will find it useful to be able to coerce A/B test values, particularly
in testing and sometimes (for CS purposes) on the live site.

2) I do not see a convenient method of having Ms. Smith get the same website
at home and at her workplace. That may or may not be a problem for you.

3) You may find "control" and "experiment" to be limiting once you start doing
this frequently -- they aren't nearly as self-documenting six months down the
line when you're staring at your web page wondering "What possessed me to make
this button teal?!"

4) I'm pretty sure you have a race condition for creation of new tests. (This
has bitten A/Bingo a few times.)

~~~
tomjen3
I do not see a convenient method of having Ms. Smith get the same website at
home and at her workplace. That may or may not be a problem for you.

How did get this to work with A/Bingo? Did you "just" write the A/B tests such
that they are tied together with the user database?

~~~
patio11
A/Bingo has a notion of "identity" (a string which is unique among users of an
application). If you don't have one, you get one assigned randomly. Your
identity gets associated with your account as early as possible, and if you
sign in on another computer, I overwrite that computer's identity with yours.
(Merging would be cleaner, but this works in practice for my site.)

The way A/Bingo picks alternatives guarantees that someone with a particular
identity will always see the same alternative for a particular test. See the
source code for details.

Since identity management is abstracted by the A/B testing framework, you can
make it as simple or complex as you want, and it hooks into user management
with on the order of ~5 lines of code.

~~~
tomjen3
Nice, thanks.

------
sdrinf
Cog is a little webdev framework of mine, a work-in-progress steal from
spending couple of years in different companies.

Since A/B testing is considered pretty fundamental to most of my clients, I
thought I'd share with the dear community of HN one possible implementation of
it. You can see A/B testing in action on the sample page:

[http://github.com/waverz/cog/blob/master/cog-
guestbook/pages...](http://github.com/waverz/cog/blob/master/cog-
guestbook/pages/guestbook.html)

When used together with scriptor, an A/Btest is about 1 line in the design
file (db rows, and counters are automagically set up on first page visiting).
Stand-alone you might want to integrate it with a templating engine of your
own :)

Code is PD, and review is more, than welcome.

~~~
Nycto
> review is more, than welcome.

You opened the door, so here is my feedback:

\- You never have to explicitly define any of the super globals as global...
you can delete any line that says "global $_SESSION;", for example

\- The database interactions look like they are heavily dependent on global
state. It is also a tightly coupled dependency

\- Use __construct for constructors instead of the class name:
<http://www.php.net/manual/en/language.oop5.decon.php>

\- Foreach is better than a for loop for iterating over arrays

\- Single character variable names only make code more confusing. You write
code once, you read it a dozen times.

I challenge you to rewrite this with only using classes (no global functions)
and to write unit tests for your code. Remember, if you have to jump through
hoops to test your code, you need to rethink what you're doing.

------
aliasaria
As an alternative, I wrote a port of Vanity for PHP that we use for A/B
testing. It uses redis to store data

<http://github.com/aliasaria/PHP-redis-A-B-Testing>

~~~
armandososa
This is an honest question: is REDIS the best option for storing A/B tests or
is just a matter of taste? MongoDB or even MySql could work?

~~~
patio11
I try to keep most of the A/Bingo work in a cache store (Redis, by choice)
because sites with traffic two orders of magnitude or so higher than mine
would blow up if they used MySQL for it. At something like Facebook game
scales I think it would be smart to take A/Bingo and rip the database out
entirely.

Writes hurt. A/B testing requires, minimally, O(tests * participants) writes.
(And more reads, but the disk will be on fire from writes first.)

However, despite the cost, A/B testing benefits minimally from ACID. If you
lose 2% of writes, oh well, it comes out in the wash anyhow. Thus the huge
hardware/maintenance budget you'd need for web scale A/B testing on an ACID
architecture makes little sense.

