

How Optimizely Is Improving Website Personalization [video] - pg
http://www.bloomberg.com/video/how-optimizely-is-improving-website-personalization-f4yaOT9oRGeXCTvnD7mPQA.html

======
jcr
pg, any chance you'd be willing to swap out techcrunch reporters for bloomberg
west on the next demo day?

------
ntaylor
Optimizely has a somewhat distorted perception of "statistical relevance" in
terms of test results. When we put Optimizely up to the test, we found that
their platform was declaring a "winner" long before sufficient data had been
collected. There's nothing specifically wrong with Optimizely, but it can, at
times, paint an incomplete or otherwise misleading picture.

Regardless of what Optimizely's reporting dashboard tells you, you should
always verify the data yourself to ensure that the result is accurate.
<http://www.evanmiller.org/how-not-to-run-an-ab-test.html>

Optimizely is a really excellent tool and it provides an essential service to
our company. Prior to Optimizely, we used an in-house a/b engine which
required constant support by technical resources. With Optimizely, the company
can make snap decisions and test them immediately.

~~~
maheshs
<http://visualwebsiteoptimizer.com/> is also a great tool to A/B testing. I
have never used Optimizely and not sure about the comparison, but
visualwebsiteoptimizer was sufficient for me for not to look into another tool
;).

PS: I am not associated with visualwebsiteoptimizer or their team. I was just
a happy user year ago.

~~~
ntaylor
We also spent considerable time testing VWO and found it to be lacking in
certain areas that were key to our business. Both products are quite good, but
there are some subtle differences that made us stick with Optimizely in the
end.

