Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Testing involves paying a price in example counts in order to buy analytical power at some unknown rate. Insignificant results arise when either (1) people react with a lot of variance and that power/example rate is small or (2) the effect you're trying to analyze is so tiny that you need a great deal of power to see it at all.

Unfortunately, it's difficult to determine the difference between those two cases. If many of your tests are failing to be significant, it could be that you're simply never investing enough to get the power you need: your users are somewhat inflexible to the design changes you're making.

It could also be that many of your changes simply are pretty insignificant and by the time you wait a full year gathering simply infinite numbers of impressions, you'll find it was just a waste of time.

Again, practice statistics with a great deal of self-awareness. They're only meant to inform.



Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: