Hacker News new | past | comments | ask | show | jobs | submit login

I don't think this is statistically significant.



I disagree, I think it's statistically significant. I ran a quick z-test:

p0 = 34

p1 = 45

n0 = 8004

n1 = 6622

z = -2.0924

p = 0.03662

So, the improvement is statistically significant at 95% confidence (p < 0.05)

As far as practical significance, that's debatable . . .


A z test relies on the normal approximation, no? I don't think that is appropriate with proportions so close to 0.


Hi, I'm the author— this is 98% confidence.


I think people are probably more concerned that it doesn't have economic significance. It might have statistical significance, but the effect is still very small, so does it really matter? That's a common trap people forget to consider. You see this a lot in finance research where some variable is statistically significant in a model, but the difference in the economic outcome is so small that it doesn't matter.

In this case, these are indeed tiny percentages. But you're going from something that a lot of people dislike and that is more complicated (from a technical standpoint), so we can simplify things and user interaction with live chat is not impacted.


You'd hope that the percentages here would be small: people need to be facing a problem with your product before they are part of the sample population.

There is a lot of potential economic impact here; during a major problem (which WILL happen) the number of people looking for support would increase sharply. If people are blind to your support system, you could be saying goodbye to a large amount of customers. Ignoring stuff like this is a far more common trap. We spend a lot of money on firefighters, even though we hope they never have any fires to fight.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: