

A/B and Qualitative User Testing - geeko
http://blog.slicedbreaddesign.com/index.php/2009/05/ab_qual_testing/

======
paraschopra
Hey guys, a shameless plug but my startup is creating a product that tries to
take A/B testing to the next level. It combines A/B testing with multiple
goals, WYSIWYG editor, visitor segmentation, powerful analytics and funnel
analysis to provide a very powerful environment for testing and targeting.
Plus we provide developer API for testing, analytics and targeting.

Here is the link to the product tour: <http://www.wingify.com/learn-more.php>

Buzz me at paras@wingify.com if anyone is interested in trying out the
product.

~~~
danielhodgins
Went to your site, and couldn't find any info on pricing. As a startup guy
that's pinching pennies I could probably handle a $19 or $29 monthly
subscription for a testing/metrics service that's better than Google Website
Optimizer, but anything more than that would cut into my ramen budget. All the
features you have listed here are great, but I care more about when you will
be launching in beta, and how much the service will cost.

~~~
paraschopra
I agree that the pricing is not visible on the website and the reason for that
is that we are still in the price discovery mode. You gave a perspective on
pricing which is very useful in this regard.

Another reason of not having pricing on the site right now is that we have two
possible paths to go forward: pure product selling OR a providing complete
solution (product + testing strategy + implementation). The latter one leads
to variable pricing which is very client dependent.

Anyway, let me know if you want to try out the solution. I'd be happy to
provide you with an account.

~~~
DenisM
Methinks you are hiding form customers. Not displaying any clue on pricing is
the best way to not get any feedback. Unless you are overwhelmed with prospect
and have a good reason to hide from new ones, I suggest some soul searching in
this regard. If nothing pops, refer to "customer development" by Steven Blank.

------
erlanger
Maybe I'm ignorant of the proper way to do A/B testing, but I shy away from it
because of the prospect of presenting users with inconsistent/volatile
interfaces.

~~~
patio11
This is an implementation issue.

Assuming you are talking about presenting users inconsistent interfaces in
terms of User Bob seeing both the A and B, then this is not too hard to solve.
If your issue is with evolving your interface rapidly enough that User Bob
would see A1, B2, A3, A4, and A5 in the period of two weeks, then that's a bit
outside the scope of this discussion, but it is a pace-of-change issue rather
than strictly speaking an A/B testing issue.

Anyhow, back to Bob. When we first see Bob on our site, we assign him a random
identifier which is presumed to be unique. Should Bob log in or otherwise
demonstrate his identity to us, we associate the random identifier durably
with the identity. Any other anonymous user who signs in as Bob will
henceforth have their random identifier overwritten with the canonical Bob
identifier.

Observe that this means anyone who we know is Bob will always have the same
constant random identifier, from here on out.

OK, now for each A/B test we do, we take a unique identifier of the test (the
database primary key, or the name if we can guarantee names are unique, or
what have you) and concatenate it with the user's identity, then hit that
string with a good hash function. MD5 works fine in practice: it doesn't have
to be cryptographically secure, it just has to provide good entropy.

We now have a big number which is entropic but which will always be the same
for a given user/test pair. Splendid: take the modulus of the number of
choices, and give that user the Nth option.

Thus, as long as Bob logs in prior to seeing the A/B test, Bob will always see
the same alternative for that test, until you cancel the test.

Now, if users do not log in, it gets a wee bit trickier. You can cookie them
the first time you see them, store identifiers in a session, use their IP
address (or hash of it) as an identifier, or use other tricks, but it is
fundamentally difficult to follow humans across computers without them taking
affirmative steps to declare their identity to you.

If this explanation leaves you scratching your head, and you'd prefer seeing
it in commented code, I suggest looking at the source code and usage docs of
A/Bingo. ( <http://abingo.org> ) Disclaimer: I wrote it.

~~~
btilly
The hash trick is clever. But what happens when you start with 4 versions,
then find that one version is worse than the others? Do you have to start the
test over? Is there some kind of rebucketing algorithm you use?

~~~
patio11
I'd just start the test over. If you want to do rebucketing, see the following
Ruby-esque psuedocode:

<http://pastie.org/610630>

