Optimizely has a somewhat distorted perception of "statistical relevance" in terms of test results. When we put Optimizely up to the test, we found that their platform was declaring a "winner" long before sufficient data had been collected. There's nothing specifically wrong with Optimizely, but it can, at times, paint an incomplete or otherwise misleading picture.
Optimizely is a really excellent tool and it provides an essential service to our company. Prior to Optimizely, we used an in-house a/b engine which required constant support by technical resources. With Optimizely, the company can make snap decisions and test them immediately.
http://visualwebsiteoptimizer.com/ is also a great tool to A/B testing. I have never used Optimizely and not sure about the comparison, but visualwebsiteoptimizer was sufficient for me for not to look into another tool ;).
PS: I am not associated with visualwebsiteoptimizer or their team. I was just a happy user year ago.
We also spent considerable time testing VWO and found it to be lacking in certain areas that were key to our business. Both products are quite good, but there are some subtle differences that made us stick with Optimizely in the end.
Regardless of what Optimizely's reporting dashboard tells you, you should always verify the data yourself to ensure that the result is accurate. http://www.evanmiller.org/how-not-to-run-an-ab-test.html
Optimizely is a really excellent tool and it provides an essential service to our company. Prior to Optimizely, we used an in-house a/b engine which required constant support by technical resources. With Optimizely, the company can make snap decisions and test them immediately.