

Referly Homepage - New Layout 16 Hours After Launching A/B Test - dmor
http://blog.refer.ly/referly-homepage-new-layout-launches-16-hours

======
ericclemmons
Even though most A/B testing tools will tell you a variation has "won" in a
short period of time, I highly recommend running tests for longer periods of
time (few days minimum, ideally a week).

\- Consumer/user behavior is different day to day, and often follows trends
(like weariness to commit or purchase prior to holidays)

\- Even with statistical signifance, I've often seen the winner flip on a
daily basis because of traffic shifts.

\- The tool doesn't know if your traffic sources are consistent (but you
should!)

These aren't concrete by any means, but neither is a short-ran multivariate
test. If your winner will be up for over 16 hours, give it some more time to
"bake" than that :)

