

How to determine if your A/B test is statistically significant - zackattack
http://blog.asmartbear.com/easy-statistics-for-adwords-ab-testing-and-hamsters.html

======
bmalicoat
I like this rule of thumb a lot. It's very straight-forward and easy to
remember so you can apply it quickly to see significance and then use a more
rigorous method if necessary.

------
jsm386
There are tons of tools that can test statistical significance for you:

For example, <http://www.splittestcalculator.com/> or
<http://www.usereffect.com/split-test-calculator>

------
steveplace
_But what about the vast majority of people who don't click either ad? That's
the "ad impressions" that didn't lead to a click. Shouldn't those count
somehow in the statistics?

No, they shouldn't; those are "mistrials."_

I don't think those are mistrials. Clicks vs impressions is a separate test
and should be treated as such.

~~~
dkokelley
The test only counts if an even amount of impressions is given. The viewers
aren't given a choice between ad A and B, they have a choice between A (or B)
and nothing. I don't think the author explained this well enough. Obviously
(to everyone here) the test is inconclusive if one ad is shown much more than
the other.

------
wmeredith
We can do this for you at the click of a button at 360voltage:
[http://360voltage.com/blog/2009/09/ab-split-testing-out-
of-d...](http://360voltage.com/blog/2009/09/ab-split-testing-out-of-dark-
ages/) (For Free)

</self promotion>

------
hadley
The statistics are a little bit off - the Chi-squared isn't appropriate for
the example he gives (sample size too small), although it would be fine in
practice. What you really want in that situation is the exact binomial test.

~~~
smartbear
Good point! Although with around N > 25 you're OK. See:
<http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm>

------
araneae
And besides that, she's clearly leading the witness (Hammy). See the outtakes:
<http://www.youtube.com/watch?v=VmZ7p3ExIc8>

------
Torn
So, he's just explained Standard Deviation?

~~~
showerst
You'd be amazed how many smart people don't have a decent grounding in
statistics, so any easy reference to explain this is nice.

~~~
Torn
Fair point, and yes it's quite informative from that point of view. I didn't
like his approach however, he seemed to be beating around the bush a lot with
phrases like

    
    
      I'm here to rescue you with a statistically sound yet incredibly simple formula
    

than giving a more honest 'hey, what you need is Standard Deviation, and
here's how it works!'

~~~
smartbear
No Torn, you're still not using the correct statistic.

THIS HAS NOTHING TO DO WITH STANDARD DEVIATION.

Also, how can you say that the article isn't "honest," especially given the
complete, precise explanation for the stats at the end.

~~~
Torn
Seems I'm haemorrhaging karma over this. I misjudged the method used
(Pearson's Chi Square, as pointed out by hn user pibefision). Fine, that'll
teach me to skim read and comment.

My point above was that the author could have come out and labelled it as such
in the article.

