
Google seems to have broken email forwarding - wglb
http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-email-forwarding/
======
reidrac
It's been years since I last did some serious email hosting, but it looks like
the SPF rule is the problem.

Google is forwarding mail for dnalounge.com but the SPF rule doesn't allow
Google's SMTP servers to do that: "v=spf1 a mx ptr ~all". That could explain
why the email gets in the spam box; failing SFP should increase the "spam
score".

Besides I think SOFTFAIL shouldn't be used in production; and I also doubt
that Google should be taking seriously a SOFTFAIL anyway; so please take this
comment with a pinch of salt.

SPF is easy to get wrong, and it always backfires at you ;)

EDIT: seems that I may be right, according to this comment
[http://www.jwz.org/blog/2015/03/google-seems-to-have-
broken-...](http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-email-
forwarding/#comment-160734)

~~~
jedbrown
Gmail's "send as" feature doesn't work that way.

 _" A@dnalounge.com is logged in to GMail Web Client as A@gmail.com and sends
a message to B@dnalounge.com. Google's SMTP servers deliver that to
cerebrum.dnalounge.com with an envelope sender of A@gmail.com. (THIS IS WRONG
ON SO MANY LEVELS.) cerebrum turns around and bent-pipe forwards back to
Google's SMTP servers, who determine that Google's SPF record doesn't list
cerebrum as a designated sender for gmail.com (given the preserved envelope
sender of A@gmail.com)."_

[http://www.jwz.org/blog/2015/03/google-seems-to-have-
broken-...](http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-email-
forwarding/#comment-160756)

~~~
agwa
In my >5 years experience running a mail server through which gmail users send
authenticated mail, that is not accurate. To confirm that nothing has changed,
I just looked at the mail logs, and in the last 24 hours I don't see a single
instance of gmail doing what is described. In every single case, the envelope
sender address is user@mydomain, not user@gmail.com.

~~~
vacri
Are the users' gmail accounts 'gmail.com' or 'mydomain'? Because I've sent
mail from servers using a google mailbox to authenticate, and the from field
was overwritten with the name of the mailbox used.

~~~
agwa
The users' gmail accounts are at user@gmail.com, but they're sending mail as
user@mydomain, via mydomain's authenticated SMTP server. This is exactly the
scenario described in the comment at jwz.org.

~~~
vacri
I read it as the opposite of jwz. You say your users are sending mail via
"mydomain" SMTP. jwz says his users are sending mail "from their phone up to
Google's SMTP server".

The goog then sends mail to the receiver at 'dnalounge' MX, where it's then
shuffled back to the receiver's gmail mailbox. I don't read his setup as
having the sender involved with the dnalounge SMTP server at all.

------
jmount
Obviously good email (deep in a conversation) being declared "spam" has been a
problem with my business for some time also.

I figure Google takes all email in a 2 step process (either intentional or
accidental)

    
    
       1: get a good fraction of the world on GMail
    
       2: intermittently declare any non GMail mail spam (
          greatly lowering the utility of non GMail).
    

This isn't about any mass mailings or even mailing lists. This is about
directly replies from me to people I have been emailing for months all of a
sudden ending up in their Google Spam folder. Combining that with Google's
good reputation and it comes off like "my dog ate my reply" (even though the
mail is in their spam folder, the point is it is easier to convince a Yahoo
user something is wrong on their side, than to convince GMail user something
is wrong on their side). And it isn't that I all of a sudden decide to include
a lot of links, attachments or zip files.

And for the "I don't like conspiracy theories crowds." From the first this
said "either intentional or accidental". It would be enough for this to be a
fortuitous bug for the effect to hurt non GMail users disproportionately.
Google could even fail a fraction of GMail and the overall image would still
be "email is flakey in general, we'd better all switch to the service provider
for safety."

I known spam filtering is hard- but GMail has some really strong signals that
I would think could dominate here (like incoming email contains text unique to
a recent outgoing email). Or email is from a sender you clearly have a
relation with.

(note: "Google takes all email" is a possible outcome, not a plot/conspiracy.
A bug like this on a system of this size can have a big impact. Of course that
does mean somebody running a system of this size might have an extra
responsibility to look out for such things.)

~~~
jimrandomh
For some reason, gmail classifies nearly all of the transactional email I get
from Amazon (order acknowledgements, shipping notifications, etc) as spam, and
no matter how many I catch and un-mark, it won't stop.

Spam classification is a hard problem, but I don't think they've put the
false-positive/false-negative tradeoff in the right place, and they really
need some safety valves like offering to whitelist an address when you un-mark
it as spam.

~~~
ibz
I always mark Amazon mail as spam. Could it be that if a lot of people mark a
certain kind of email as spam, Google just treats it as spam?

~~~
ccozan
I would not recommend this. Amazon emails can be unsubscribed. And they are
not really spam, they are not coming unsolicited, aren't they?

~~~
thaumasiotes
I get a ton of unsolicited email from Amazon. It tends to be of the form
"order confirmation for product XXXXX" or "your order of XXXXX has shipped". I
don't view this as particularly unreasonable, though.

~~~
nitrogen
Emails in response to you direct action (placing an order) are the very
definition of _solicited_ email.

~~~
thaumasiotes
No, emails that you ask to receive are the definition of solicited email. When
was the last time the grocery store emailed you confirming that you'd just
paid for your groceries?

~~~
johnduhart
> When was the last time the grocery store emailed you confirming that you'd
> just paid for your groceries?

Never because my grocery store isn't a website.

~~~
thaumasiotes
OK. When was the last time facebook emailed you to confirm that you liked a
comment?

~~~
cptn_brittish
The last time I had Facebook emails before I marked Facebook as spam every
action which somehow concerned me on Facebook resulted in me getting a email
and consequently flooding my inbox. Admittantly this was a few years ago when
I finaly had enough of it so they might of gotten better but at one point they
did.

~~~
thaumasiotes
Yes, I picked facebook as an example specifically because of their reputation
for emailing you about everything. But I think you'll find that they tend to
email you when other people do something that, in some sense, concerns you,
not when you do something. (Remember the ancestor comment defining "solicited
email" as email that responds to an action you take.) "This person just liked
your comment" emails are plausible. "You just liked this person's comment"
emails are self-evidently absurd, but they fall perfectly within the
(spurious) definition of "solicited" that nitrogen wants us to believe in.

------
copsarebastards
I've long supported a proof-of-work concept augmented with whitelisted keys
for email spam filtering.

There are basically two cases here:

1\. The sender is sending one email to one receiver. The sender computes the
proof of work and sends it along with the message. This takes some time, but
typically it can be done in the background. Waiting few seconds between
sending and receiving an email is typically not an issue, and in fact many
email clients wait for some time before sending anyway to allow "undo send"
functionality. Rather that doing this computation on the mail server, it
should be done on the sender's client, so that the server doesn't get
overloaded with proof-of-work computations.

2\. The sender is sending many emails to many receivers (a mailing list). In
this case when the receiver signs up for the mailing list, the sender sends a
request for a whitelist token to the receiver's mail server. IF the user
accepts the request, the receiver's mail server returns the token and then the
sender sends all further communication with the token instead of with a proof-
of-work. This solves a few problems: a) Receivers opt _in_ rather than opting
_out_ of receiving communications. The ubiquitous pre-checked "send me spam"
checkbox loses its effectiveness. b) Receivers can revoke tokens at any time.
c) Senders who are sending large amounts of legitimate mail don't have to
compute a proof-of-work for every email they send.

Receivers simply drop email which doesn't come with either a proof-of-work or
a whitelist token. This drives up the cost of sending large numbers of spam
emails because each spam email requires a large amount of computation. And
even in cases where a spammer has a large amount of computation at their
disposal (botnets are a common case) it makes it easier for servers to
distinguish between mailing lists and spam: a large number of identical emails
could be either, but mailing list mailings should come with a whitelist token.

Doing things this way would mean we can drop these terrible DKIM and SPF
systems that both fail to prevent spam and make it difficult to send
legitimate mail.

~~~
tie_
> Doing things this way would mean we can drop these terrible DKIM and SPF
> systems that both fail to prevent spam and make it difficult to send
> legitimate mail.

I can understand your feelings about SPF, but why is DKIM a terrible system in
your opinion?

~~~
copsarebastards
Because we still have spam and it's difficult to send legitimate email. It
doesn't achieve its goals and gets in the way of achieving other core goals.

------
reuven
I have been running my own SMTP server for about 25 years. I finally decided,
in the last month or so, that it just wasn't worth the effort involved in
handling spam and anti-spam stuff -- not only from Google, but from other
providers who are so worried about spam that they falsely tag all sorts of
stuff as spam. I found that accurate configuration and maintenance, including
of SPF and DKIM, wasn't worth my time and effort.

And so, I have now moved my e-mail to Rackspace. I had to use their chat-based
support several times while migrating my e-mail to the new server, and it was
truly fantastic. On that point alone, I feel fortunate.

The $20/month I'm paying Rackspace is a pittance compared to the time and
effort I was spending trying to keep my old SMTP/IMAP servers secure, as well
as the false-spam tagging that happened all-too-often.

It sounds like jwz wants to have his cake and eat it too, and I sympathize.
But I'm not sure that it's possible any more to spend a non-trivial amount of
time configuring e-mail servers, tinkering with them such that they'll work
with big companies (and especially Google). The Internet is no longer the
simple, fun playground that we old-timers remember, and that effectively means
giving control over some services to people who are paid full-time salaries to
take care of these inter-connectivity issues.

~~~
kuschku
For me it luckily works – I only host a dozen of users on my server, but it
works nicely, and – after having spent a few months setting it up once – also
runs nicely with Google and the other large providers, even Yahoo and Hotmail
(for now).

------
xkarga00
They enjoy breaking things in general

[http://googleonlinesecurity.blogspot.ca/2014/04/new-
security...](http://googleonlinesecurity.blogspot.ca/2014/04/new-security-
measures-will-affect-older.html)

[https://support.mozilla.org/en-US/kb/thunderbird-and-
gmail](https://support.mozilla.org/en-US/kb/thunderbird-and-gmail)

 _Gmail accounts created on or about June 2014 (exact date unknown, Google
only mentions "second half of 2014" in their new authentication blog post)
won't work with Thunderbird until the Thunderbird team implements Google's
non-email-standard authentication. This is currently scheduled for Thunderbird
38 which will be released on April 7, 2015. See bug 849540 for the full
technical details_

------
Touche
I honestly can't figure out what he's attempting to do here.

It sounds like he doesn't want to force a work email account on his employees.
I don't think many people would have a problem with a work email, it's the
norm after all, but fine.

But he also doesn't want to add their personal email addresses to his address
book (is this why he wants the @dnalounge.com addresses?)

I'm not sure what this setup is attempting to accomplish.

~~~
rottingchris
He does create a work email account for his employees. He doesn't want to
force them to use a separate email client so he allows his employees to set up
forwarding from his server to their personal gmail account. They can use his
smtp server from gmail for sending stuff out so they get to use both their
personal and work email from the same interface.

~~~
Touche
I've never used an email client that didn't support multiple accounts.

So again, I'm not sure what problem he's trying to solve here. To be clear I'm
not saying he shouldn't be able to set up his email this way and have it work,
just that I don't get why he WANTS to do it this way. What is it
accomplishing?

~~~
pat2man
Gmail.com is this email client. You can't add an external account, you have to
forward.

~~~
fps
gmail.com supports fetching email from external servers via POP (it's under
settings, accounts). His users could use that and skip most of gmail's spam
filtering completely. It doesn't (reliably) support forwarding mail to it
without using some list software in the middle. I believe if jwz set up
mailman or something similar, it wouldn't have a problem with that, because
mailman would properly re-write the envelope sender as the list address.

------
sschueller
At least Google puts the email in Spam.

With office 365 and hotmail email just goes into the abyss. We have SPF and
DKIM yet some mail just never arrives when sent to hotmail. The server
responds with "250 queued for delivery" but the mail never arrives and doesn't
go into spam either.

Very annoying when order confirmations don't arrive.

------
lucb1e
Google has always had issues with legitimate e-mail the spambox, but then
again, I have yet to see a spam filter with no false positives. Google just
errors a lot more on the side of classifying email as spam compared to most
email services (more false positives, but less spam in the inbox). What makes
this even more annoying is that they hide the spam folder so that users don't
even know that it exists anymore. I always have to give directions where to
find it and invariably get reactions like "Oh, I didn't know GMail had a spam
box!" or "Ah thanks, I would never have found that."

I did consider setting up a route in my mailserver to GMail using my Google
account credentials for their SMTP server, but then decided they should better
get their own shit straight. It wouldn't scale to do this for everyone.

~~~
sergiosgc
Gmail is awful regarding false positives. People discount it because: a) it's
a free service; b) false negatives are more frequent and thus more annoying.

What is difficult in spam classification is having no false negatives _and_ no
false positives. Any decent CS graduate can create a spam classification
system with no false negatives given the leeway to introduce false positives.
That's what Gmail did.

~~~
lucb1e
> b) false negatives are more frequent and thus more annoying.

I think this is the case for the user. For the sender it's a major issue and a
huge annoyance, but you don't measure user satisfaction by asking senders.

------
spullara
Gmail will dump emails from google.com into my spam folder if I have them
forwarded from another gmail account. It is infuriating that they can't even
solve this problem within their own domain.

Example of a message from Google, to a custom domain gmail account, forwarded
to another custom domain gmail account — found in my spam folder:

[https://www.dropbox.com/s/uwxjdji8n6g6jfo/Screenshot%202015-...](https://www.dropbox.com/s/uwxjdji8n6g6jfo/Screenshot%202015-03-05%2012.00.31.png?dl=0)

------
Alex3917
The first thing to check is whether the DKIM and SPF checks are actually
passing. Email forwarding tends to break them, and if that's what's happening
then there are specific things you can do depending on what exactly is
breaking.

------
shiggerino
Gmail is horrible in so many ways. Complete disregard for the MIME structure
of the emails. All parts that aren't text/plain or text/html gets bumped to
the bottom, even if they are clearly declared as inline. And then there's the
infernal top posting they force on you. To get to one attachment I had to page
through a wall of unsnipped irrelevant garbage, if they want to force an
obnoxious posting style, at least put the attachments right below and together
with the top-level message, not all the way at the bottom.

------
gchokov
These guys.. is it just me, that think that Google miss some serious traction?
Fundamental things like Email Forwarding not working now? Is everything just
hit and miss there at Google HQ?

~~~
inigoesdr
One guy out of millions having problems forwarding email is indicative of
everything being hit and miss across all Google projects? You and the OP might
be jumping to some unjustified conclusions.

~~~
gchokov
No. Just look at the number of projects they dump to the trash can, compared
to number of successful ones. Litarally... hit and miss.

------
raverbashing
I've seen emails with the google.com domain end up in the spam folder (yes, it
was legitimate)

So, yeah

------
Zezima
This was happening to me as well. I was trying to get an email from a known
individual, but after being sent 4-7 emails, none came through. I checked the
email service which the emails were being sent to and they were no longer
there (forwarded already), and also not in my gmail inbox yet.

They're still no where to be found, might be in limbo, but email forwarding
definitely took a hit yesterday

------
mpnordland
I run my own mail server, and I've found it to be pretty much pain free. Of
course I'm the only one who uses it, and I don't do fancy stuff, but after a
few bumps when starting out, it works great. I've only had to fix things
occasionally, like heartbleed. On the whole, it's been really nice. Is my
experience abnormal?

~~~
gvozd
I've had the same experience running my own pair of iRedMail servers in
FreeBSD jails. It was easy to set up, and not hard to secure. I haven't had
trouble with spam so far, and it has allowed me to migrate off of gmail.
Setting up failover took a little work, though.

------
mzs
Jered has this figured-out: [http://www.jwz.org/blog/2015/03/google-seems-to-
have-broken-...](http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-
email-forwarding/#comment-160756)

------
DanielBMarkham
I was looking into using email for rich content delivery yesterday as part of
this "Watch me program" thing I'm doing. How difficult could it be to send
rich html to somebody on email?

Turns out, freaking difficult. For one of the simplest protocols in the world,
email is now a byzantine disaster. Different clients render things
differently, different providers block or don't block things -- and that's not
even getting into trusted domains and spam issues, which you end up depending
on third parties to assist.

Add into this mix Google's notorious black-box/impenetrability/lost-in-space-
customer-support? I don't see an optimistic future for you.

I'd rather go back to hand-coding COM in C++ that become a network email
engineer. That's got to be a brutal job. You're dependent on so many moving
pieces and you control so very little.

~~~
frozenport
Do you think the strict restrictions reduce spam, and phishing attacks, and
email client hacks? For example, many clients dont let you change the default
colorbof a hyperlink.

~~~
DanielBMarkham
I think you'd be better off with an open standard, rich, dynamic html that
runs wherever the user is. Then put all the sophistication in whitelisting
senders. We also need to bolt-in secured messaging so that nobody except the
people corresponding have any idea what's in the message.

The way it's set up now, with hundreds of vendors implementing different
control protocols and tens of thousands of spammers trying to break in? I
don't see anybody being happy -- some spammers get in anyway, and email users
have different experiences for the same dang net service. Admins want to pull
their hair out. It really is a miracle things work as well as they do.

If we can't get email working right? It doesn't portend for a bright future
for the rest of internet traffic.

------
fifthesteight
I've been blaming my hosting company, getting everyone in their support chain
to take a crack at the issue. I always just assumed that gmail was the working
side of the problem, not the problem itself.

------
wahsd
I wonder if it is related to a change I recently noticed in how email address
suggestion and autocomplete, i.e., suggestions, work that behaves differently
than before. It's almost like they deemphasized frequent user or maybe domain
accounts or something. The effect is that the first suggestion is no the
address you are most likely looking to use, and thus risking erroneously
addressed emails. So take care.

------
nvk
Yahoo also broke it, none of our customers get @coinkite.com emails for over 2
years now.

We simple prohibit @yahoo emails to be added as contact and use Mailgun.

------
netheril96
Why don't you setup the reverse? That is, let their GMail act as a user agent
to receive and send emails on behalf of the company account.

------
aembleton
I've got email forwarding set up for a domain on 1&1.co.uk and don't have any
problems on gmail. Others who have mail forwarded from the domain don't have
any problems on hotmail or yahoo either.

Maybe Google gives extra trust to 1&1; or maybe there is some other thing that
they check for.

------
hudell
And to make it worst: Google don't run filters on messages that goes to the
spam folders, so you can't even use that.

I always wanted to do the opposite: Run a filter on the spam folder to delete
emails that I know for sure that really are spam, so I wouldn't need to delete
them manually.

~~~
pavel_lishin
I'm not sure what you mean; do you mean that their search bar doesn't include
spam filters? (I assume that's what you mean since filter creation is sort of
built into the search bar.)

~~~
hudell
You can search in the spam folder, but some filter actions won't run on
messages that are there.

------
1ris
>What do I have to do to make Google stop fucking me?

Stop using it. You already have your own email server.

~~~
jmount
No, you would have to make everybody else stop using it.

~~~
blfr
In this case, only his employees. Because this is the issue, not general
deliverability.

~~~
cwyers
Right.

And, well, I am sympathetic to the argument that Google shouldn't do things
the wrong way. But you really have no leverage to make them. Google offers the
ability to use Gmail for e-mail sent to your domain, that's Google Apps For
Business (or I guess Google Apps For Work). If that's unacceptable, you either
have to deal with Google not caring about your needs (unless you're paying
them, you're not the customer anyway) or do without Google.

------
RIMR
Anyone else wondering why JWZ feels the need to complain when he is clearly
using Gmail in a way that Google advises you not to?

If he wants to use Gmail for his business he needs to get Google Apps. Having
your employees use their personal e-mail accounts for business is completely
unprofessional.

Why even bother running an in-house mail server if all it does is forward
e-mails? Cut the costs and pay for the services Google offers for business and
get your employees more focused by keeping them off their personal e-mail on
the job.

------
Steko
Why is it every time I see an article critical of Google it has invariably
been vote flagged down?

------
TazeTSchnitzel
Are you sure it's forwarding? Do other domains have the same issue?

------
uptown
What's the best solution for spam filtering if self-hosted emails?

------
cwyers
This sounds like the old vaudeville routine about "Doctor, it hurts when I do
this." Well, don't do this.

------
jbeda
I'm late to this thread, but hopefully I can add some value.

I've recently been through this for my own personal email. I have a domain
with some interesting rewrites to the address that I forward to GMail. I put
together my set up into a Docker container:

[https://github.com/jbeda/docker-postfix-
forwarder](https://github.com/jbeda/docker-postfix-forwarder)

It is worth breaking this down into each direction -- sending and receiving.

 __Sending __

This is the easy side. You simply run an SMTP server and have GMail use this
to send. If you want others to not mark you as spam you 'll want to look at
setting up SPF (publishing which IPs are allowed to send from your domain) and
perhaps DKIM (digitally sign your email).

GMail used to allow sending from aliases directly without an external SMTP
server but that is disabled. Old accounts are grandfathered in.

 __Receiving __

This is where things get complicated. Essentially, you are having folks send
mail to your server and you are turning around and relaying that email to
GMail. GMail has a hard job here. It doesn 't know if it should trust you and
that you are acting on behalf of the user or if you are an open relay sending
spam. I've found after I change my forwarding set up I have to police my spam
folder for a couple of days to retrain GMail.

The real complication here is SPF. Sending domains will publish their own SPF
records. When doing this they can either specify a soft or hard fail. If they
specify a soft fail (`~all`) then there is a chance that GMail won't mark it
as spam. But if they use soft fail (`-all`) then it will go to spam all the
time. The problem is that the sending domain (say evite.com) doesn't list your
relay IP as having permission to send that mail and so GMail respects that.

Having GMail __pull __the mail via POP is one solution here but that
introduces latency.

Or you can rewrite the "envelope sender" so that you are honest about the mail
coming through your server. The accepted scheme to do this is SRS. This is not
a silver bullet though. If you are forwarding a lot of spam, GMail may decide
that your SRS domain is spammy and penalize all incoming mail.

Also, if you are forwarding a lot of spam, GMail will throttle you. It'll have
you back off and wait to forward more mail. Your best bet is to find ways to
eliminate obvious spam before you forward to GMail.

My solution seems to be working okay for now, but it is a pain the ass and I'm
honestly not sure it is worth it.

EDIT: Here is a tutorial that I used to inform my approach. It is worth
digging in to. [http://seasonofcode.com/posts/setting-up-dkim-and-srs-in-
pos...](http://seasonofcode.com/posts/setting-up-dkim-and-srs-in-postfix.html)

------
callesgg
Well this dude's email setup seams insane on so many levels to me.

------
tls
1: email is hosted offsite, yet reliant on Google/Gmail to do in house
work/intermail.

2: when you are reliant on an external source to solve a problem within your
own house you are not prepared.

3: solution: reduce reliance or point of failure by either bringing it all in
house and forwarding the remains or bring root to the source and source from
root and use mask and forwarding.

~~~
rev_bird
I'm confused -- are you saying doing your own email hosting is the way to
avoid getting stung by Gmail spam filters? Seems like an "out of the pot and
into the fire" kind of situation.

~~~
tls
confusion is the root of the problem, either remove the offending host and do
it all in house or move yourself within the root of the problem and continue
conducting business.

he could bounce off another provider/server so he can still maintain what he
is doing now - he just has not explored any other solution except blaming. the
arguement "it should just work" is not vaild if you are reliant on another
service.

------
calinet6
"I run my own mail server, but..."

Whelp, there's your problem.

~~~
rev_bird
I get why it's hard to trust mail from some random IP from Amazon Web
Services, but I'm very wary of taking big chunks of the internet and just
saying, "well, people aren't allowed to do that anymore, just the huge
corporations." One of the best parts of the web is that it's decentralized.

------
epochwolf
> Google seems to have broken email forwarding

Programmer misconfigures SPF record and blames google instead, news at 11.

