A couple of things I forgot to mention in the post.
1. The numbers are based on hits through URLs with tracking information. Some visited and voted from the Twitter feed, some hit the base URL. This affects absolute numbers, obviously, but the %-ages should be about the same.
2. I also tallied up negative and positive votes, thought that perhaps people on Monday were more stressed and negative than on Sunday. But there were too few dislikes, so it's hard to make any conclusions -
like !like
Sat 22 3
Sun 44 3
Mon 34 1
Tue 34 1
Wed 35 4
EDIT - I added raw emails/opens/votes data at the end of the Tumblr post, just in case.
There is email campaign software available that tracks the open times on previous emails to each person on your list. The next time you send a newsletter to that person, the email is sent at the optimal time for the person to open, read and interact with it. I forget what the software was but I always thought it was a great idea.
While at Bronto software (http://bronto.com/), one of the features I worked on was just that. We called it Send Time Optimization, and from what I understand, it's not an unheard-of feature amongst email software providers.
Completely off topic — you worked at Bronto? I did a short (like 1 or 2 day) contract to write a first pass at their Ruby client gem a couple years back. Seemed like nice guys. I get shivers thinking about Ruby and SOAP though. Eesh.
If we treat "Sunday" as the baseline and use "votes" as the count of a successful email, it appears that you can say with a somewhat high degree of confidence that Sunday is better than Saturday, but nothing else.
One other thing I think you should experiment with is time of day. There was another great post (not able to find the link yet, will update when I do) that talked about open rate being highest within the first few hours of sending regardless of the send time.
I've always been a fan of sending between 3-6am, although I think I just took this as a rule of thumb from someone I worked with an never questioned it much. Would love to see a similar study on send times within the optimal range of days.
MailChimp has a neat page where they list the average open rates for the 5.5 million mailing lists people manage using MailChimp's platform: http://mailchimp.com/resources/research/
"Most emails are sent between Monday and Friday, with the highest volume on Tuesday and Thursday. More subscribers open email, however, on Wednesday and Thursday."
"Subscribers are likely to open email after 12pm, and the most active hours are 2-5pm."
"Your newest subscribers are your best subscribers. They’re most engaged immediately after signup, and then trail off over time."
Those are averages though, your mileage may vary. For the newsletters I send out, Tuesday morning has proven to be the sweet spot.
What time of the day you send the email makes a big difference. Especially if your list is large. ISPs do magic with large sends to prevent spamming. If you send on a Sunday evening, and others send Monday morning at 3am for example, most people's email view is reverse chronologically ordered. So on Monday morning, your email is likely to be on the bottom of the pile.
I put the "engagement" clicks into excel and looked at the chi-square test to determine the likelihood the data differs significantly from a random distribution: http://depts.alverno.edu/nsmt/stats.htm
Expected range: 36.2 (average of the clicks)
P-value: 0.14
Typically you look for p-values of less than .05 to indicate statistical significance. So unfortunately this one test suggests the data falls within the range of normal (random) distribution.
Your margin of error is 4.38% [1], which makes the comparison between all non-Saturday meaningless. However, it seems like the numbers for Saturday vs other days are useful.
To answer your specific question, if you had 10x more data, your margin of error (with 95% confidence) would drop to 1.39%, so yes, you could reasonably get fairly different results.
I was disappointed to read that you skipped Thursday and Friday. In my domain, Mondays are the peak of activity, slowly sliding down to a bottom on Friday/Saturday. I would have liked to see your numbers for every day. Next time! (In that case, your MoE would have been 5.37%, btw.)
I am too very curious about Fridays, but I suspect it would require getting the time of the day right too. Send it too early, and it's pre-weekend morning stress. Too late - and everyone's already gone for the day.
PS. I don't know why I picked (mod 5). Makes zero sense now.
Because your sample is so small, the results are easily skewed by a few people. As the sample size increases you'll get a more representative picture of what is happening.
It doesn't invalidate what you've found, but does mean you should now continue on with your experiment to increase the sample size - and help ensure your data wasn't impacted by something else - e.g. vacations / a conference that was attended by many of your subscribers / etc etc.
Yes, if you re-ran the experiment 10 times, it could be completely different. The presented data is based on a single campaign.
Perhaps something was happening that Saturday, something that the post failed to take into account. Perhaps something happened during those weekdays, causing more engagement than normal.
"I went to kindergarten, I know how the alphabet works!" :)
Interestingly enough, nothing was happening on that Saturday. That I actually checked.
I obviously don't disagree that there's a lot of variables in play and some vary faster than others. What I meant by "10x" was that if the list were 20k subscribers instead of 2k for this particular experiment - what would've been the chance of the Saturday open rate jumping up.
are the engagement numbers for sunday correct? it appears to be highest, which doesn't match the conclusions.
[edit: maybe i'm not understanding, but engagement on sunday is highest at 47, yet the final recommendation is for "Tuesday or Wednesday" (35 and 39). why wouldn't you choose the day with highest engagement?]
[also, incidentally, those numbers are all the same to within a couple of standard deviations (1 sd ~ sqrt(n) ~ 6). so the sample size is too small to draw any really firm conclusions.]
It looks like the conclusion is overly simplistic (but it's a blog entry based on a single experiment, so it's quite reasonable to limit it to simple conclusions)
If I understand the data correctly, Sunday has a low open rate, but of those that open, it had very high engagement rate.
The author seems to be saying that the open rate is low enough to make Sunday a "bad day", but that really depends on what your email is about.
If getting people to read the email is your priority, then avoid Sunday, but if you mostly care about getting them to respond in some way, then Sunday could be good - those that read it will have the free time to actually respond.
(Of course, while the limited data provided by this experiment is interesting, it's probably not worth basing any serious decisions on)
What I wanted to share was the fact that there was a noticeable difference to begin with, so it's something worth paying attention to.
The exact pattern will depend on the audience. If it's VmWare emailing IT people, then the open rate is probably going to be the same through the workdays and zero on weekends. If it's some hobby stuff, then it'd be in reverse. For my particular project, it just happens that Saturday sucked, which is a good thing to know.
I certainly think that your experiment has value - my apologies if it came across any other way.
- If nothing else, it shows that there is a difference, and the day you pick for your newsletter matters.
- It seems that it would be helpful for other people to repeat your experiment (but stretch it to 7 days) to see what the results are for their readers.
- The difference between the open rate and engagement rate is quite interesting.
It's actually reasonable. This is an opt-in mailing list and subscribers are generally interested in the project, so if they are hit with a relevant update on their free time, they will respond.
(edit) The rationale behind "Tue/Wed" is that I value views more than votes. Agreed on std.dev., yet Saturday still stands out as the least performing day.
A couple of things I forgot to mention in the post.
1. The numbers are based on hits through URLs with tracking information. Some visited and voted from the Twitter feed, some hit the base URL. This affects absolute numbers, obviously, but the %-ages should be about the same.
2. I also tallied up negative and positive votes, thought that perhaps people on Monday were more stressed and negative than on Sunday. But there were too few dislikes, so it's hard to make any conclusions -
EDIT - I added raw emails/opens/votes data at the end of the Tumblr post, just in case.