
You are Not Smarter than A/B Testing - danielodio
http://go.DanielOdio.com/optimizely
======
darkxanthos
I love the passion here, and I love Optimizely (we just recently signed up
with them). Keep running you're bound to end up somewhere awesome. There are a
few key points here that make this latest test you're running flawed though
(and I know from running my own 120 multi-variate test).

Running all of those tests in a single run will help you find a test that
improves over the original almost definitely. What that large test doesn't
tell you definitively is how those variations perform against one another nor
does it tell you the likely increase of your measured metric.

Furthermore, running tests without adherence to a rigorous thought process
(where each variation essentially is a separate hypothesis, with possibly
unique covariates, and different reasons why your test might be flawed.

Your own intuition and gut need to play a part in the tests so that you can
understand the results and so that they don't cause more confusion.

In the end, you will end up with results that you could've gotten to 50x
faster by just testing the variations you believed had a reasonable cause to
do better (just a couple rather than 1000). This is again due to the fact that
Optimizely is only telling you that it's likely the winning variation would do
better and the fact that the individual results don't compare to anything
aside from the original page.

I'd love to chat more and share my love for data and the knowledge I've gained
as I've traveled this exact path with you. Reply if you wanna email/skype
about this a bit.

