

False premises common in anti-A/B-testing arguments - noelwelsh
https://medium.com/designing-for-results/aadbe3465dc4

======
ehutch79
Some of these fall premises, are actually admitted to be true in the
article... wtf?

Also, a lot of the quotes are not so much against a/b test, but people who
claim a/b testing is a panacea, and are recommending the wrong tools to the
wrong people. (ie tools based around static content to people with entirely
dynamically generated content)

basically the biggest problem with a/b testing is that some of it's most noisy
proponents have no idea what they're talking about. (this is true of a lot of
things)

------
dylanrw
The biggest problem I have with A/B or Split testing is that it makes it easy
for inexperienced people to assume there are multiple right answers to every
problem and that they all warrant 'discussion' or testing... Most often this
is not the case, and people who jump straight to A/B everything are the sort
who don't really have/trust any experience and would rather spin wheels and
throw spaghetti at the wall.

That said, sometimes there are two (or more) options that warrant testing (say
a marketing strategy) and then it works great.

Test strategy, don't test everything, don't test design principles.

~~~
adw
The biggest problem I have with _not_ A/B testing is that it breeds a culture
where people assume they really know what's going on.

Common sense is often only common sense in retrospect!

------
varelse
I've always thought the problem with A/B testing is that it's mostly hill-
climbing optimization which is likely to hit and get stuck at a local optimum.

~~~
wikwocket
A/B testing doesn't necessarily mean "test two subtle shades of the same
thing." For a great example, see the Highrise A/B test:
[http://37signals.com/svn/posts/2991-behind-the-scenes-ab-
tes...](http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-
part-3-final)

This is a good example that disproves many of the false premises mentioned in
the OP, as the clear winner of the A/B tests was a bold dynamic design with
lots of character, that was completely different from the original designs.

~~~
anywhichway
Great example to bring into the conversation. One complexity to consider is
that the long form versus the smiling person may have retained completely
different user demographic. To me, the smiling person may represents an easy,
idiot proof service with customer support, but may be lacking more advanced
technical features, which is fine if you are set up to cater to that demo.
Technical services like Github may suffer from increased sign ups from the
wrong demo if it means their forums are swamped with beginner level questions
that they aren't equipped to hand hold.

------
greglindahl
This would be 100x more useful if it was "Avoiding common pitfalls of A/B
testing" \-- since the author admits that poor usage of A/B testing makes most
of the premises true.

~~~
jfarmer
His point is that these outcomes aren't inherent in the process of A/B
testing, not that they aren't consequences of certain types of A/B testing.

