
An email list’s open rate drops over 50% after changes to Gmail spam filtering - iamben
https://detailed.com/audience-halved/
======
ryandrake
As much as I'd like to reduce my reliance on "cloud" services, Gmail is one of
those that I just can't quit, since they've pretty much solved the spam
problem for me. I'd rather a few false positives get filtered than the deluge
that comes with inadequate (self-hosted) E-mail.

Sorry if this guy's "audience building" business is affected, but as an end
user not worried about "my open rate", I appreciate Google's aggressive
filtering and urge them not to let up.

~~~
roblabla
Urg.

The thing is, I run my own personal email server. It's not used for spam
__ever __. It 's only personal email to family members, friends, and the odd
reach out to support. The volume of email is so small it's ridiculous. I might
send up to 10 mail (1 receiver each) per month. I set it up with SPF and DKIM
properly, triple checked.

And yet, for whatever reason, _sometimes_ my email end in spam. Generally when
I send attachment (like pictures). But sometimes it's just completely
arbitrary.

How about this : for you it might be a "few false positive", because so much
of the wanted mail traffic originates from google in the first place. But for
the few of us that use mail _as was intended_ , this becomes a huge problem.

Having a mailing list doesn't necessarely mean the content is unwanted. I'm
subscribed to HN replies for instance. I _want_ those email. If it started
going to spam, I might consider "hey it's a few false positive". But for the
HN replies guys, it would be maddening to be unable to fix the problem because
Google can't be arsed to have a support channel.

From my POV, Google is way past "aggressive filtering" territory and way into
"monopolistic behavior".

~~~
mabbo
> I set it up with SPF and DKIM properly, triple checked. And yet, for
> whatever reason, sometimes my email end in spam.

It's probably your IP address.

Sure, SPF and DKIM say you are allowed to send these emails, you are who you
say you are, etc, but you also aren't a Sendgrid IP address (or someone
similar) so you're treated as less trustworthy.

I could be wrong, but I believe many email servers take these things into
consideration (please let me know if I'm mistaken!)

~~~
gregmac
If you're literally running an e-mail server from your cable/DSL connection,
you're much more likely to be marked as spam, regardless of doing everything
else correctly (SPF/DKIM/reverse DNS), because there are lists like SORBS DUHL
[1] and RBL-DNS DUL [2] that index these. Some servers block entirely, others
just use it as an indicator that your mail is more likely to be spam.

Many spam filtering systems work on a 'points' system, so being on the dynamic
IP list adds a lot of points, and if you also have other things that add
points (being a mailing list, having 'unsubscribe' links, attaching images,
having links) it might be enough to put it over the edge and get marked as
spam.

You can do a lookup [3] to see if you're on any of these lists.

[1] [http://www.sorbs.net/faq/dul.shtml](http://www.sorbs.net/faq/dul.shtml)

[2] [http://www.rbl-dns.com/dul.shtml](http://www.rbl-dns.com/dul.shtml)

[3] [http://multirbl.valli.org/lookup/](http://multirbl.valli.org/lookup/)

~~~
CaptSpify
Although what you say is true, I've been running my own mail server out of my
house for years with proper spf/dkim/dns and have never had problems of my
mail being sent to spam.

------
paulsutter
That's hilarious: A newsletter about how-to-spam is treated as spammy and the
author is upset. In the end, everyone thinks they're one of the good guys.

> The aim of this blog is to help anyone with a website find proven content
> ideas that they can use to drive more traffic and links to their own sites.
> The concept is already working, with one recent article helping a reader get
> on the front page of Product Hunt.

~~~
SamUK96
This is instantly what I thought too. Essentially this guy is teaching how to
get rich by using SEO, catchy gimmicks/ideas, and the likes. The spam teacher
themselves get filtered, how ironic.

~~~
martokus
Yes, welcome to the world of Viperchill - the shadow king of get rich by
teaching others how to get rich schemes. I wouldn't be surprised if Google
personally targeted him, I would do :)

~~~
viperchill
Sadly this is happening to a lot more people than just me otherwise I would
not have written about it.

------
threepipeproblm
I only skimmed this so forgive me if I missed something. But could this simply
be what happens to you when a sufficient number of email recipients classify
you as spam, and you then reach some critical threshold?

>> I changed my sender email address to see if would help

But if the pattern of content got flagged as spam by Google users, the sender
address might not be the only determinant.

>> I was relieved to discover that others with newsletters were having similar
troubles very recently.

But if this results from crossing some sort of user-classified spam threshold,
one would probably always be able to find some recent "victims".

This is not intended as a criticism -- I'm sensitive to Google's seemingly
arbitrary power -- just an idea to consider or rule out.

~~~
ballenf
Similarly, it could have been 3-4 consecutive newsletters with particularly
low engagement that put him across some threshold. Not people marking as spam,
just not opening or opening and quickly deleting.

Personally, I can't remember the last time gmail marked anything important
spam. Maybe a newsletter whose absence I hadn't even noticed, but nothing
actually important to me.

I just wish I could get gmail spam filtering on my cell phone. Robocalls are
out of control -- 4-5 / day. (I've looked at some of the apps / subscriptions
for this, but I'm not ready to give a relatively unknown 3rd party access to
my contacts or to adding contacts, which they generally require.)

~~~
kemitche
Try making google voice your main number? They had solid robo filtering if I
remember right. I'm on Fi now and I can't remember the last time I received a
robo call.

------
IanCal
Google sends me reports on the spam we get to one of our mailing lists.

I hadn't seem them in a while and found that they themselves were being sent
to the spam folder.

edit - I'd forgotten about this actually, and checked, it happened again
today.

It's apparently happening because people send me spam, so I mark it as spam.
Because I've marked things sent to that email address as spam, they're saying
their own report is now spam. The description of the problem is sounding more
python-esque as I continue so I'll stop there.

~~~
charred_toast
Can confirm — Google's own Spam reports are going to the Spam folder. Was it
the word Spam that triggered it? lol.

------
victorNicollet
The spam problem has _never_ been solved, no matter how good spam filters may
have become in the recent years. It's still an arms race, with a front that is
moving one way or another every few months. Even today, a spam filter is a
piece of software that you trust to throw away e-mails without letting you
read them, based on its own decision that you don't want to read them.

It seems insane to me that people would willingly let a program decide to
throw away inbound mail based on unspecified rules. And I nearly start
frothing at the mouth when someone suggests that, since their software decided
to throw away my e-mail, it is somehow my responsibility to fix that. It
isn't.

If there were thousands of different anti-spam solutions, and one of them
behaved badly, then senders wouldn't care, and users would vote with their
feet (or wallet, or app store) until the behaviour was fixed. But since Google
has a near-monopoly on e-mail, it's now up to the senders to do the work and
jump through the badly-documented ever-changing hoops.

~~~
draw_down
Do they really have a "near monopoly" on email? How near are we talking here?

~~~
saul_goodman
It's scarey stupid how much the "big 3" own email now. Google, Yahoo, Hotmail.
When any of them change up their spam fighting techniques it causes ripple
effects across the entire email ecosystem now.

Have Yahoo mail subscribers on a Mailman mailing list? Not if you're not
running the latest Mailman which has a new workaround just for them. Yahoo
started bouncing legitimate mail a couple years ago due to the normal way
mailing lists have ALWAYS handled sender addresses, since before Yahoo
existed.

I get it, sometimes desparate times call for desparate measures (for fighting
spam). But jeeze, it's super annoying to have to break your package manager to
get an upstream mail server to support one user who uses one certain mail
hosting service.

------
Nazzareno
It's not a Gmail issue!

By using an ESP (like Aweber, Mailchimp, etc) means that you are sharing part
of your "reputation" with thousands (or even millions) other senders. So even
thou most of the ESP are quite efficient in cleaning-up their customer base
from spammers, the more clients they have, the more it's difficult.

Here you can find an pretty accurate and updated analysis of ESP's clients
"quality": [http://mainsleaze.spambouncer.org/may-2017-in-spamtraps-
esps...](http://mainsleaze.spambouncer.org/may-2017-in-spamtraps-esps/) Aweber
is #6, which is not a good signal.

That said, you may switch ESP by choosing one that provides a you custom
"identity" such as dedicated IP, DKIM, PTR, SPF records (which usually means
quite a lot of setup and costs) or choose an ESP that doesn't have tons and
tons of clients, eventually with a free version.

[http://www.emailvendorselection.com/](http://www.emailvendorselection.com/)
is an updated ESP directory. You are of course also welcomed to try mine:
[http://www.mailup.com](http://www.mailup.com)

~~~
tjoff
" such as dedicated IP, DKIM, PTR, SPF"

DKIM, PTR and SPF really ought to be enough, right?

~~~
Nazzareno
Yes, but you can't have a custom PTR without a dedicated IP.

------
mikl
When I saw the friend was using Mailchimp, it made a lot more sense.

I get lots of spam from Mailchimp (ie. people subscribing me to their
newsletter, without my consent), and complaining to Mailchimp about it seems
to have no effect.

So since Mailchimp allows bad actors to send mail with their service, it’s not
entirely unfair that Mailchimp mail gets flagged as spam.

~~~
gk1
The author uses Aweber, not Mailchimp. Besides, Mailchimp is quite strict with
their opt-in policy and they don't hesitate to suspend accounts that ignore
it.

~~~
tyingq
One of the author's peers uses MailChimp and saw the drop.
[https://detailed.com/wp-content/uploads/2017/07/paul-
jarvis....](https://detailed.com/wp-content/uploads/2017/07/paul-jarvis.jpg).
That's in the article.

 _" Did you read the post"_...that always shuts down discussion.

~~~
gk1
You're right, it's unnecessarily hostile. I edited that out.

------
kazinator
Gmail is not magically going to white-list recipients for receiving some spam
that they opted into on some random website it knows nothing about.

People sometimes opt-in in good faith, and then when they get the e-mail, they
realize it is garbage. They tell their e-mail client that it is junk, rather
than going through the ritual of unsubscribing. Probably, in may cases, they
probably don't even make the mental connection between what they received and
the site they visited where they checkboxed something; they don't recognize
that what they're receiving is something they signed up for. Their reaction is
"WTF is this crap?" and report it as spam 500 milliseconds later without
giving it another thought.

If enough users flag the posting as spam, Gmail will probably learn that and
help the remaining users.

------
mrbunnyman
After reading this I checked my spam folder. Found an email from one of my
best friends from highschool, who I've sent 100s of emails back and forth
with, letting me know he would be in town in my spam folder... Well guess I
didn't want to see him over the holiday weekend. GAAHAHAHHAH.

See as how a lot of my friends use their own domains / hosting for email I
guess I'll be spending a lot of time in my spam folder.

~~~
rstupek
What's really bad is your friend had no idea you never saw his email because
it ends up in a spam folder and not bounced back instead

------
danvoell
Google could get really evil in this regard. But then again Facebook has been
doing this for years. "We didn't think the content was relevant to this
audience. Pay us some money and we will allow your email to reach more
people..."

~~~
AznHisoka
This is already the case today. I get so much irrelevant mail in my main inbox
that I ignore a lot of legitimate email in the process. And you can buy Gmail
ads which appear in the top of your inbox.

------
II2II
Perhaps businesses are becoming too agressive at requesting email addresses,
people are providing fake email addresses, and the recipients are responding
by flagging it as spam?

I bring this up because I started receiving suspiciously legitimate spam a few
years back. No red flags were raised by the email itself (e.g. no indications
of phishing), the companies exist, the recipients exist, everything was
geographically consistent. It was as though certain individuals were providing
my email address as their own to businesses yet not to friends or colleagues.

When you consider how agressive some of these websites are with newsletters, I
would not be surprised if people were providing fake email addresses just to
get rid of the prompts. I would not be surprised if Google started flagging
those newsletters as spam, even if the odd user genuinely wanted them.

------
alyandon
The hilarious part about Google's spam filtering is that I have crons that
send logs in emails to my own address using Google's own smtp servers (that
require authentication with my own credentials) and they end up getting placed
in spam.

------
creo
I bought electronic content that was sent to me via signed-email and it got
into spam ... somethings not right.

------
coding123
All the spam I get is newsletters from the perspective of the spammer. F that.
My email should be for people.

~~~
eli
Well, if you didn't sign up for it then it probably is spam. But lots of
people want to get newsletters and if you want to get it, then it isn't spam.

~~~
Spivak
Even if you did sign up for it it's probably spam. That's the real issue here.
Given how aggressive businesses are with signing up people for unwanted
newsletters it seems completely natural for Google to be suspicious of them.

~~~
ReidZB
Yep. I waste an hour (probably) every week unsubscribing from various
newsletters and marketing campaigns that I don't care about. Most corporations
have largely taken "this person has given me an email address" \-- even if one
is necessary to use their service! -- as free license to hit them with all
sorts of marketing messages. Really, these _are_ spam -- unsolicited, unwanted
email. It's just that "spam" has taken a stronger, more negative meaning.

------
77pt77
Some years ago I got a reimbursement from google regarding expenses they were
paying me. I though they had totally forgotten about it and contacted them.

Even the google representative knew their own financial related emails were
systematically being tagged as SPAM on gmail.

That's how aggressive it was.

I couldn't stop laughing for minutes.

------
dwringer
I've had emails from family members go straight to gmail's spam folder on two
different occasions, and a third occasion in which the mail appeared to have
sent according to the sender's mailserver, but never showed up anywhere in my
gmail account. From what I could tell glancing around some forum posts, mine
was hardly a unique experience and is reliably solved (well, in the first two
cases) only by explicitly whitelisting every single address from which you
expect to receive correspondence. I'm glad to see more attention to this issue
of overzealous filtering.

------
brihas
I work for a small non-profit and we've had the same issue. We use SendGrid to
send out our email newsletters and the open rates have fallen from ~27% to
~13% over the past month.

Does anyone here have contacts at Google who might be able to help? [Edit:
I've also reached out to SendGrid support to look into this]

Looking at the data more, the drop is due to a ~73% drop in Gmail opens.
Hotmail and Yahoo opens remain stable:

1) See drop in overall open rate here:
[https://www.dropbox.com/s/j8grg3r684zshhz/Drop%20in%20open%2...](https://www.dropbox.com/s/j8grg3r684zshhz/Drop%20in%20open%20rate.png?dl=0)

2) See drop in Gmail open rate vs. Yahoo and Hotmail here:
[https://www.dropbox.com/s/0fdvbddwno27mgf/Drop%20in%20Gmail%...](https://www.dropbox.com/s/0fdvbddwno27mgf/Drop%20in%20Gmail%20opens.png?dl=0)

~~~
dhekimian
Since you didn't provide your domain name, I can't troubleshoot for you.

I recommend reading my comment
[https://news.ycombinator.com/item?id=14720344](https://news.ycombinator.com/item?id=14720344)
and following the troubleshooting steps I outlined.

~~~
brihas
We use the news.pariyatti.org domain for these emails. If you're willing to
help troubleshoot, that would be much appreciated. Otherwise I'll try to work
through your recommendations, although a lot of it is very new to me so I'm
not sure how far I'll be able to get. You can reach me via email at my HN
username @pariyatti.org if you wish to discuss further.

------
hamstercat
In a previous job, my team was responsible (among other things) of the
company's newsletter. Nothing great but nothing spammy either, and we made
sure to follow anti-spam law by having the newsletter subscription being opt-
in and having a clear and easy unsubscribe flow. We didn't send the emails
ourselves, but rather used a big player in that area.

What I learned is that sending emails and not getting treated as spam by any
of the big email providers is an absolute mess of tinkering and hoping it'll
work. Not all of them would categorize our emails as spam, but Outlook.com
started doing it out of the blue, and even a year later we didn't have a clue
as to why or how to make it stop. Support for email senders is as good as for
users, meaning getting anything else than a canned response wasn't happening.

------
anoetudh3tnh
I have zero sympathy for these people. My spouse for example often gets added
to mailing lists when purchasing a product (just like we all do). Instead of
clicking the unsubscribe link, they'll just "open" the email without reading
it to mark it as read in their inbox, and immediately go on to the next one.
These are essentially spam that my spouse is too lazy to mark as such. I have
to wonder how many of the "opens" the author's lists get are just like this.
"Oh, that stupid list I got subscribed to somehow. Mark as read." I have a
very deep suspicion that they're mailings are essentially spam to a lot of
people who receive them and this is correct classification by Google.

------
Ardren
Just a anecdote, but this month there was a significant phishing attempt
emulating an Australian electrical provider which got through Gmail's spam
filter. Gmail has now updated their filters to detect it, but it's now
catching genuine email as well.

That said, it's been a long time since Gmail has had a false positive that I
was surprised about (i.e. not a spammy newsletter that I subscribed too)

------
dhekimian
DETAILED.COM has a bunch of issues that would easily trip SPAM filters.

\-------------------------------------------------

TL;DR: DNS, SPF, DKIM & DMARC are not properly configured for DETAILED.COM.
Fix and test the issues below.

\-------------------------------------------------

 _Start here:_
[https://mxtoolbox.com/domain/detailed.com/](https://mxtoolbox.com/domain/detailed.com/)

1\. SPF Records

 _Issue: You should only have 1 SPF record but you have 2.

Combine them. [https://www.spfwizard.net/](https://www.spfwizard.net/) _

Current:

    
    
      detailed.com.           3600    IN      TXT     "v=spf1 include:smtp.groovehq.com ~all"
    
      detailed.com.           3600    IN      TXT     "v=spf1 mx include:send.aweber.com -all"
    

Should be:

    
    
      detailed.com.  IN TXT "v=spf1 mx include:send.aweber.com include:smtp.groovehq.com ~all"
    

2\. You SOA record is incorrect

Your current authoritative DNS servers are:

    
    
      detailed.com.           172800  IN      NS      ns.sourcedns.com.
      detailed.com.           172800  IN      NS      ns1.sourcedns.com.
    

Your current SOA is:

    
    
      detailed.com.           3600    IN      SOA     ns.liquidweb.com. admin.liquidweb.com. 2017042600 86400 7200 3600000 14400
    

In your SOA, ns.liquidweb.com is incorrect. It should be ns.sourcedns.com

3\. Missing DKIM records

Check if AWEBER is sending the correct ones.

Use the steps outlined here
[https://www.phishingscorecard.com/](https://www.phishingscorecard.com/) to
find your DKIM key and test to see if it's working properly.

4\. Setup DMARC to get reports on your failed email

 _If you had set this up, you would be getting notifications from Gmail,
Yahoo, etc as to why your email is not being delivered properly._

I suggest doing a 14-day free trial via
[https://dmarcian.com/](https://dmarcian.com/) Once you create an account,
they will provide you with RUA & RUF addresses and process the XML responses
for you. Don't forget to add the _DMARC TXT record to your DNS.

5\. Test your email using [https://www.mail-tester.com/](https://www.mail-
tester.com/) This site is fantastic and will help you narrow down on actual
delivery issues.

------
raverbashing
The core of the issue seems to be an anti-phishing feature that Google seems
to have dialed too high in some cases

But yeah, communication channels are getting more and more clogged with
irrelevant communications (not even span alone) so it's getting tough to get
across the barrier

------
wolco
Gmail is not a good provider if you care about seeing all of your emails all
of the time.

It does a good job getting rid of spam but if you need to check the spam
folder for missed emails then it makes sense to leave the spam in the main
inbox and save a few clicks

------
eli
Huh, weird. I haven't noticed this and I send a lot of email newsletters.

------
zippergz
I wonder if they've started handing SPF or DMARC misalignment differently.
I've noticed a lot of those kinds of problems with newsletter-type mail in the
past.

------
RichardHeart
Good thing everyone uses gmail right? Centralize everything under one company!
Centralize all the things! I use gmail :(

------
nilsocket
If Google wasn't able to find the problem, then they probably should rollback.

They can't hurt their customers this way.

~~~
snuxoll
You say that like Google cares, over the years they've amassed a large swatch
of consumer email users (probably due to Android) - they have basically gotten
to a position where they set the rules and if it hurts you tough luck.

Email deliverability is a hard problem these days, spam has caused us so much
grief that large email providers are doing really annoying things in their
spam filters to combat the problem but I've noticed more and more legitimate
email is getting caught in the crossfire. Unfortunately, I don't see this
being fixed anytime soon - and it's personally making my life increasingly
difficult as someone who had to give up self-hosting his email and switching
to a hosted Zimbra service just to stop getting flagged by Google as spam
(even though I passed DMARC/DKIM and SPF validation and had a non-blacklisted
IP).

~~~
kuschku
> as someone who had to give up self-hosting his email and switching to a
> hosted Zimbra service just to stop getting flagged by Google as spam (even
> though I passed DMARC/DKIM and SPF validation and had a non-blacklisted IP).

There is a trick how you can get out of the spamfilter: Send mail, regularly
(like a few mails per week) to friends/family that use Googlemail, and tell
them to mark your mail as "not spam". After roughly a month, you’ll have
gathered enough points that Googlemail won’t mark you as spam anymore.

------
jordigh
They're measuring "open rate" by putting links or images in the email that
have uniquely identifying information, right?

As much as I'm frustrated with Gmail's spam filtering (I also use a personal
email server, and it's annoying to get my personal email filtered by Gmail), I
absolutely loathe seeing that tracking information in links, even if it's from
a mailing list I specifically subscribed to. Often times they're newsletters
that are also available on the web. I'd rather just go to ddg to search for
the web version to read.

Of course, that's probably a case of
[https://xkcd.com/1105/](https://xkcd.com/1105/)

------
davidgerard
tl;dr have your SPF, DMARC and DKIM in perfect order, 'cos Gmail has started
filtering _hard_ on those.

