

Top 7 split testing blunders you must avoid - paraschopra
http://visualwebsiteoptimizer.com/split-testing-blog/split-testing-blunders/

======
jaysonelliot
I'm on board with every statement except the first one, "Test whispers instead
of screams."

The author suggests that you should only test the big things that could be
"grand slams," like a "radically different design."

There are two major problems with that statement. First, a radically different
design is going to have so many differences that you'll never be able to
pinpoint which element made the difference. If you're seeing a 10% lift, does
that mean you might have gotten rid of something that was working great, and
you could have had a 30% lift by just fixing the actual problem? What if the
radical new design actually lower sales? Now you have to find out which of the
forty-seven changes caused it, or just go back to the old design altogether.
That's a lot of unnecessary and expensive work.

The second major problem is that by going for "grand slams," you're basically
throwing spaghetti against the wall to see what sticks.

Instead, take the time to be analytical and form real testable hypotheses
about what could make an impact.

Jared Spool tells a story in the book "Web Form Design" about a checkout
process that started with a simple two-field input form asking users to
register for an account or login before paying for their purchase. After
looking at user data carefully, they found that almost all of their dropoffs
were happening right there. They tried moving the login fields to the end of
the checkout process instead of the beginning, and saw a 45% conversion
increase, over $1.5 million in the first month.

A radical redesign wouldn't have fixed the problem - they needed to do user
path analysis and give the issue careful thought before deciding where to make
the change.

It's often a single usability issue that's causing the problems, and you can't
find those by changing everything on a page and hoping for the best.

~~~
paraschopra
I agree. This is a guest post on our blog but my opinion is that one should do
a lot (and lot) of small A/B tests coupled with a couple of big, large scale
tests. Interestingly, most of our A/B testing case studies have been about
small changes leading to large conversion rate changes:
<http://visualwebsiteoptimizer.com/case-studies.php>

Although "grand slam" tests are essential so that you don't get trapped to
local minima.

------
tnorthcutt
(I left a similar comment on the blog as well)

“Let’s just say you get a 10% bump. Well… if you then roll that out into the
other 5 products, you can count on a 10% bump on each of those as well… giving
you a total of 50% in total increased conversions spread out throughout your
site.”

That’s incorrect. A 10% increase in conversions across all products on your
site gives you a total increase of 10%, not 50%. Perhaps #8 on this list
should be “incorrectly interpreting results”.

------
micheljansen
I think a more important blunder is failing to understand statistics.

"I’ve ran plenty of tests where one variation was the control, hitting 95%
confidence, after just a handful of conversions ... Because you need to run
your tests long enough to get accurate results! You need to get past what I
call “random coin syndrome”."

Statistics don't work that way. A result is either significant (e.g. you can
either draw conclusions with a 95% confidence rate), or it is not. If you
reach 95% confidence after 14 conversions (10 vs 4), it either means your
questions are too broad or you failed to control for external influences.

~~~
Jabbles
The flaw is more fundamental than that. By testing the confidence each time,
you run the risk of vastly overestimating the true significance of your tests.

A previous discussion of this exact problem:
<http://news.ycombinator.com/item?id=2368825>

