

Ask HN: Do you A/B test on your sites? - f1gm3nt

Do you have some type of A/B test on your site?<p>Do you use Google Optimizer, something else, or have you coded your own?<p>If you're not currently using A/B tests on your site, why? If you are using A/B testing on your site has it been useful?
======
byoung2
At work, we used Google Website Optimizer on the redesign of one of our sites.
The site is a coupon/deal site that makes several million dollars per year
through affiliate links. The site hadn't had a new design since the late 90's.
Since 99% of traffic comes from organic search and lands on a page for a
specific merchant, and leaves through an affiliate link, we didn't want the
new design to make that process harder. Here's what we did.

The designer created 5 variations of the merchant landing page. The variations
had different color schemes, placement of coupons, etc. We used GWO to test
the original versus the new variations, using an exit through an affiliate
link as our goal. In this round, the original beat out all of the new designs
(60%+ conversions vs 30%).

Back to the drawing board. What was it about the new design that was affecting
conversions? An informal focus group revealed that the listing of coupons
looked too much like display ads, and people didn't like clicking on ads. The
designer came up with new designs that looked more like a blog. We re-ran the
tests with 5 variations, but the original still beat out the new versions
though by a smaller margin (60%+ vs 50%+). The focus groups liked the new
design better than the original, but real world users weren't clicking as
much.

That's when I stumbled on the answer. Whenever we did focus groups we showed
users the design in a conference room, and it was always on a local server.
Whenever our QA department did testing, it was always from our office, 1 mile
from our datacenter. When I sent a friend a link, he thought the site was too
slow. I suspected that since the new page was slower than the old one, people
were giving up before it fully loaded.

We re-ran the GWO tests, this time with various stripped-down versions of the
landing page with different loading times. We even created one version that
loaded even slower. Finally we had a winner: when the new design was stripped
down a little, and the images heavily optimized, and aggressively cached, we
got the load time down to about the same as the old version. This was the
version that beat out the original (60%+ conversions vs 72%). This is the
version on the live site today.

~~~
f1gm3nt
That just confirms that I need to use A/B testing lol.

Thanks for the insight.

