I put the "engagement" clicks into excel and looked at the chi-square test to determine the likelihood the data differs significantly from a random distribution: http://depts.alverno.edu/nsmt/stats.htm
Expected range: 36.2 (average of the clicks)
P-value: 0.14
Typically you look for p-values of less than .05 to indicate statistical significance. So unfortunately this one test suggests the data falls within the range of normal (random) distribution.
Your margin of error is 4.38% [1], which makes the comparison between all non-Saturday meaningless. However, it seems like the numbers for Saturday vs other days are useful.
To answer your specific question, if you had 10x more data, your margin of error (with 95% confidence) would drop to 1.39%, so yes, you could reasonably get fairly different results.
I was disappointed to read that you skipped Thursday and Friday. In my domain, Mondays are the peak of activity, slowly sliding down to a bottom on Friday/Saturday. I would have liked to see your numbers for every day. Next time! (In that case, your MoE would have been 5.37%, btw.)
I am too very curious about Fridays, but I suspect it would require getting the time of the day right too. Send it too early, and it's pre-weekend morning stress. Too late - and everyone's already gone for the day.
PS. I don't know why I picked (mod 5). Makes zero sense now.
Because your sample is so small, the results are easily skewed by a few people. As the sample size increases you'll get a more representative picture of what is happening.
It doesn't invalidate what you've found, but does mean you should now continue on with your experiment to increase the sample size - and help ensure your data wasn't impacted by something else - e.g. vacations / a conference that was attended by many of your subscribers / etc etc.
Yes, if you re-ran the experiment 10 times, it could be completely different. The presented data is based on a single campaign.
Perhaps something was happening that Saturday, something that the post failed to take into account. Perhaps something happened during those weekdays, causing more engagement than normal.
"I went to kindergarten, I know how the alphabet works!" :)
Interestingly enough, nothing was happening on that Saturday. That I actually checked.
I obviously don't disagree that there's a lot of variables in play and some vary faster than others. What I meant by "10x" was that if the list were 20k subscribers instead of 2k for this particular experiment - what would've been the chance of the Saturday open rate jumping up.
