

Why you should Stop using A/B Tests - hexgnu
http://modulus7.com/why-you-should-stop-using-ab-tests/

======
theseAntics
Popularizing testing methodologies is a good subject, but you're way off base
in terms of what A/B testing is and how it should be used. Hopefully you're
just going for a dramatic point of view to drive responses, but as a point of
professional courtesy, that's ok in your lead, not your actual content.

1\. AB is not about determining the effect of various factors in isolation.
Its about quickly reaching a solution that is close to optimal in situations
where you can't feasibly generate the kind of data you'd need to build a
model.

2\. AB is not poorly suited to dealing with "externalities," confounding
variables, etc. It is the only method that is suited to dealing with them.

3\. This does happen, but it is more of a failure in the selection of WHAT to
test, than it is a failure of the methodology itself. An understanding of
optimization strategies is helpful in avoiding this trap. In practice I think
what most people are doing is closer to simulated annealing than greedy
anyway.

Ping me @TC_Davis on twitter if you want pointers to resources on any of the
above.

