

How we split test our job listing titles with Optimizely - abreckle
http://visual.ly/blog/javascript-lovers

======
Jabbles
(Addressing the owners of visual.ly)

I hope that "Chance to beat baseline" column isn't a naive measurement of
"confidence" that's updated on every iteration.

I've mentioned this several times in threads such as these, but if you're
blindly applying basic statistical analysis to a partially completed test, you
are badly misinforming your customers.

Please read this: <http://www.evanmiller.org/how-not-to-run-an-ab-test.html>

Of course, if you are taking these results into account I apologise for
mentioning this again and wish you every success.

------
GertG
One must love conclusions like this: "Even though the results aren't
statistically significant (...), we were able to learn ..."

------
aresant
Hilariously cool. Hopefully this report is the nail in the coffin of the
"ninja" job posting fad.

Buttttt, since you're measuring clicks and not job candidates hired (the
actual important metric) is there really a relevant takeaway from the data?

Hard to quantify but if this is proper conversion rate testing you've got to
make sure you're optimizing around the proper metric.

