

How AquaSoft increased their sales by 20% with A/B split testing - paraschopra
http://visualwebsiteoptimizer.com/split-testing-blog/how-aquasoft-increased-their-sales-by-20-doing-ab-split-tests-in-multiple-phases/

======
patio11
High fives for AquaSoft and for Paras.

Guys, you can totally do this. It works. It is repeatable. Stupid little
things like button designs and calls to action and shopping cart layouts
_really freaking matter_ , despite taking 1/1000th as much time to address as
adding that next feature you're guessing some users might actually want.

~~~
bemmu
Right, but unless you run a gigantic site, can you really test stupid little
things like button designs, when you need thousands or even tens of thousands
of conversions to have enough data for a meaningful result?

I feel a bit burned, because last test I ran took me a week to set up. I
wanted to see if having extra display ads would affect the ratio of people who
convert from visitors to repeat visitors. I put new users into groups, and
after two days I had 800 people in both groups. Then later on with data in
hand I realized that the change wasn't large enough / I didn't have enough
people in my test to get a significant result.

~~~
patio11
I apologize in advance if any of this is repetitive with things I have said
before:

1) There is a difference between what you and I think are really stupid little
things and what reality thinks are really stupid little things. For example, I
think it is likely that you can (with a suitable A/B testing framework) change
a call to action on your site in under five minutes of coding, counting the
time to redeploy. Many engineers would consider that a stupid little change.
Empirically, exactly that test has resulted in double-digit improvements
before. So totally do that.

2) An A/B test which leads to the result "Not enough data to tell" doesn't
convey zero information. It conveys an important bit of information: "Well,
neither of these two alternatives lit a fire in the hearts of my users." That
should give you the ability to recalibrate your efforts in the future.

3) With the context that I run a very part-time business: for the last couple
of years I aimed at doing four A/B tests a month and getting a 5% increase out
of one of them. If the other three end without significantly significant
results, oh well. (See <http://www.bingocardcreator.com/abingo/results> \--
one in four isn't too far off what I actually get. Nota bene that is a
convenience sampling of results I've actually accumulated, not all of them.)
Of more applicability to you: you have to accept that A/B testing is a process
which converges on awesome rather than a button which is onclick:
deliverTheAwesome(). Over time, I promise you, it really does work. (It almost
can't _not_ work.) However, individual tests will frequently return null
result or tell you that the new code you just spent time writing is a waste.

4) I try to avoid doing tests that take a week to set up unless I have a good
reason to suspect it is going to seriously move the needle, because for the
same amount of work you can throw a lot of smaller things at the wall and see
what sticks. For example, I've put off an A/B test that I absolutely have to
do until I am gainfully unemployed just because of the implementation
difficulty. (A: current site selling online and downloadable software. B: what
are you talking about, thee is only an online version.) Especially if you're
new to A/B testing and don't have a good example that you can point to
yourself and say "Bemmu, you feel like quitting, but REMEMBER THIS?! Oh good
golly that worked out right. Test on!", test some stuff which takes 10 minutes
to bang out alternatives for. Conversion buttons, calls to action, positioning
of buttons, that sort of thing.

5) "Things that really move the needle" and "things that take a lot of time to
implement" do not relate to each other. In fact, they sometimes have so little
relation it is hilarious. Ask me how many weeks I have burned in developing
features no one cares about versus how many minutes it took to change five
words on my purchasing page. ("Buy a single copy via ..." -> "Get instant
access to ...") Sure, go ahead and test the button designs. Let reality tell
you what matters rather than thinking you have a good idea. I haven't the
furthest clue.

~~~
paraschopra
<http://www.bingocardcreator.com/abingo/results>

Great, I never realized you had this page! A similar page for Visual Website
Optimizer exists in the demo <http://dev.visualwebsiteoptimizer.com/?demo=1>

