
Google App Engine Silently Stopped Sending Email 5 Weeks Ago - markdown
https://code.google.com/p/googleappengine/issues/detail?id=12786
======
cramsdale
Hey Folks,

The issue reported here is linked to App Engine and Gmail tightening up their
spam filters. The root cause was an increase in organizations sharding out
their spam systems to utilize App Engine’s free tier in such a way that is (a)
in direct violation of our ToS and (b) making all of our lives suck a bit more
(raise your hand if you want spam). It’s unfortunate that while App Engine is
trying to provide a free tier that enables developers to easily use our
platform, others see it as an opportunity for exploitation. Even more
unfortunate is that it has a negative effect on legitimate users. It’s a fine
balance that has been highlighted by several users within this thread.

Spam filtering is not a perfect science, and we’re constantly tweaking things
-- with our customers in mind. This issue should be limited to new
applications where the trust signal might be a bit lower. Thus existing apps /
customers shouldn’t be experiencing issues (which was also highlighted by a
few within this thread). If this isn’t the case email me:
cramsdale@google.com. For those asking, “hey, why am I being penalized for
being a new customer?” See my previous comment about spam filtering not being
a perfect science. Then email me.

We’re here and we want to help.

\-- Chris (Lead PM for App Engine)

~~~
brianmcconnell
Long time App Engine user here. If you're sending email as part of your app,
I'd recommend using Mailgun (or similar services). You can set up DKIM,
inbound routes, etc, and generally have more control over how your mail is
processed. I never got the impression App Engine really wanted to be in the
email delivery business, so if that's important to you, there are tools that
are focused on that (and free up to a monthly quota). My $0.02.

~~~
wastedhours
+1 as well, after the changes to Mandrill recently, was looking for a new
provider and can also recommend Mailgun (at least, for ease and
configurability in set up and testing so far, not yet in production).

~~~
pritambarhate
Yes, Mandrill should have kept the free quota after DKIM and SPF verification.
Now, I will be trying out the competitors (Sendgrid and Mailgun) and if I find
them at par with Mandrill then for paid projects, I might use them rather than
Mandrill.

~~~
mrcncpt
For our clients, the appeal of Mandrill is the Mailchimp Template Editor. I
think this is where their focus is going and so if you don't need that, then
definitely check out competitors.

~~~
pritambarhate
Out of around 18 projects, in which we have used Mandrill, only in 1 we have
used the template editor. (I think only about half went for paid accounts when
the project went live.) Most of our need was simple and quick transactional
messages (Registration, Forgot Password, etc.). The main reason we chose
Mandrill was ease of set up over Amazon SES. (Even though we had AWS accounts
for all projects.)

Anyways, it's up to Mandrill to choose who they want to serve. I really liked
the service though. Best of Luck.

------
jpatokal
Google Cloud Support here. To correct the misleading title, this is _not_ a
generalized outage of any kind, but an issue with some applications sending
mail that trigger spam/malware/etc filters. Since spammers generally do not
comply with RFC 3514, filtering is unfortunately not an exact science, so
false positives happen and we're working with the customers in question on a
case-by-case basis.

Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly
recommend using these if you're planning on sending larger quantities of mail:

[https://cloud.google.com/appengine/docs/python/mail/sendgrid](https://cloud.google.com/appengine/docs/python/mail/sendgrid)

[https://cloud.google.com/appengine/docs/python/mail/mailgun](https://cloud.google.com/appengine/docs/python/mail/mailgun)

~~~
markdown
5 weeks ago your infrastructure stopped sending out mail. Our apps didn't
change, your infrastructure did.

Whether it's your outgoing spam filter that's overzealous, or if a datacenter
is on fire is irrelevant. Your infrastructure stopped doing what your docs
said it would.

Paying customers like myself have apps running on your infrastructure. Your
infrastructure stopped delivering the exact same emails that it had been
delivering for years. It did this without notification and without triggering
any errors.

Approximately 35 quote requests that should have been delivered have
disappeared into thin air. You've cost me money, and I'm disappointed to find
out that you'd been notified of this issue weeks before it began affecting me.

~~~
awjr
I really have to highlight how dangerous an approach this is. You have no idea
if the previous user of that IP was spamming. You should seriously consider
mailgun or sendgrid. They also bring a lot of features that enable you to
track when an email was read etc.

~~~
zymhan
You seem to assume a lot of people know about the ins and outs of many
different online mail delivery services. I would trust Google to do a little
better than drop emails on the floor for weeks and not say something.

------
martindelemotte
I wonder if people at Google know how awful it is to report an Appengine bug.
You have to battle for days to get your bug acknowledged.

I've noticed that they've added more support people since the early days but
it's still a pain as they seem to have an incentive to answer the tickets as
fast as possible without doing any real investigation.

The linked issue is a good example of this behavior.

~~~
jpatokal
Please remember that Google App Engine is _entirely free_ up to (roughly) 5
million page views/month:
[https://cloud.google.com/appengine/kb/#quota](https://cloud.google.com/appengine/kb/#quota)

If you're running production services, you should probably sign up for a paid
support package, which have guaranteed response times down to 15 minutes:
[https://cloud.google.com/support/](https://cloud.google.com/support/)

Disclaimer: I work in Google Cloud Support.

~~~
inopinatus
I think this is a terrible attitude. I will never implement it in my own
support model. People shouldn't have to pay me in order to report a bug in my
system. Arguably I should pay them, because one report is the tip of an
iceberg of silent affected customers.

More often than not, what most people want is simply an acknowledgement.
They're not seeking a bug bounty. And definitely not the brush-off you just
gave the GP: when someone's house is on fire, you don't demand prepayment for
the water.

~~~
pyvpx
no, when your house is on fire the property-tax funded fire department comes
and attempts to put it out.

the "free tier" of municipal services is actually just a form of socialism!!
eek!

~~~
lazylizard
adam smith. tragedy of the commons. so we pool our resources together and hire
a contractor to handle the commons for us.

------
fencepost
Reading through the thread, it looks like in the last few days it's been
narrowed down to "messages which contain the URL of the App Engine instance
silently disappear with no bounces or errors."

~~~
achant
And only sometimes. The issue resolved itself on my app two weeks ago. My
guess is there is an outgoing spam filter run amok in some of the clusters.

~~~
djsumdog
I wrote about this last year, but I feel it's still relevant. Spam filters,
especially Google and Microsoft's (whose e-mail servers don't play by any of
the standard rules nor do they send DMARC reports), are horrible over
aggressive.

[http://penguindreams.org/blog/how-google-and-microsoft-
made-...](http://penguindreams.org/blog/how-google-and-microsoft-made-email-
unreliable/)

I have seen more phishing/ransomware e-mails come through mine (lots of
malicious Javascript made to look like Excel files), so I can understand how
great the risk is. The fact still remains though. E-mail is broken and is less
reliable than an envelop dropped in a letterbox.

~~~
tacon
>especially Google and Microsoft's (whose e-mail servers don't play by any of
the standard rules nor do they send DMARC reports)

I don't know about Microsoft, but I get a couple of DMARC reports per day from
Google's email infrastructure. I also get reports from aol.com, yahoo.com,
linkedin.com, comcast.com, fastmail.com, etc.

------
tominous
Maybe I'm old-fashioned but if I was affected by this issue (and it had
persisted for so long) I would set up an SMTP server and an appropriate MX
record, then demonstrate using mail logs or port snooping that this was
definitely not a receive-side issue.

Of course the support organisation here (Google) should be well equipped to
set up this kind of testing themselves and work with a customer to root cause
the issue. In my experience though these issues are rarely taken seriously
until there is no-one else left for support to blame.

------
cyberpanther
I had a Google engineer tell me once that the App Engine email api was never
meant to send a lot of mail. That is why the quota is so low. I would
recommend using a third party service to do e-mail.

------
ajackfox
I'm pretty surprised by the long wait times between "updates" from support...
that's pretty rough. I understand it can be hard to get a good signal to noise
ratio with support, but when so many other people are reporting issues,
perhaps it's time to take it a bit more seriously?

This thread is a great example of why support is so hard.

~~~
user_0001
not really. I wish I could get away with providing such crap support as the
big players do.

If my customers don't get a reply within a few hours, then all hell breaks
loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my
time.

To be fair, when I point this out to them they do apologise, didn't realise
time difference, are in a different part of the world where Sunday is not a
day off etc. I then ask what they expect from some billion dollar companies
support wise and they say they are happy with a reply within a week. Asking
them why a one man shop doing quite a bit less than billions of dollars is
expected to provide so much more.

Anyway, rant over. Support isn't hard. You hire enough competent people as
your customers require. The end. Happy customers

~~~
Godel_unicode
Oh, is that all? Someone should let Google know.

------
waleedka
Must be an isolated issue. I run two App Engine apps and send 30K emails a day
and I haven't noticed any change in my email open/click stats, which implies
that emails are being delivered as usual.

~~~
Ensorceled
Well, that is one of the problems I kept hitting when I was on App Engine ...
issues that "only" impact 10% of customers.

------
Maxious
I've been getting bounces on Google Apps hosted gmail for calendar
notifications coming from calendar-notification@google.com

Email is hard.

~~~
pmlnr
For programmers, maybe. Running mail servers is a sysadmin task. (I mean no
attack here, this is plain statement)

~~~
tshtf
Running a mail server is simple, even for programmers.

Running a mail server with reliable outbound delivery, and inbound spam
filtering, is anything but simple.

~~~
pmlnr
A reliable mail stack is configuring postfix+dovecot or cyrus, adding dspam,
opendmarc, opendkim and a bunch of dns records.

How is this any harder than anything else in computing?

~~~
tshtf
DKIM + SPF + DMARC does not guarantee reliable delivery. Not even close.

~~~
pmlnr
Of course not. However, in the past ~7 years, while running my own mail stack,
I never had a delivery issue, though obviously this is small traffic.

At work, we handle pretty large traffic for mail, with self-hosted mail
servers as well - those are exim. No delivery issues either, although these
are only the real, important, not even remotely spam-like, core business
messages. Spam-like newsletters are done via sendgrid there.

------
awinter-py
The problem is that pesky humans can't decide what they want. Not sending mail
is a bug but not receiving mail (i.e. spam filtering) is a feature? I can
appreciate how this is confusing.

~~~
markdown
Not sending is a bug when it's undocumented, when it happens without
notification, when there's no error message, and when there's no outgoing-spam
bin.

The GAE docs say "do x and we'll send your mail". If we do x and mail isn't
sent, then that is a problem.

~~~
Godel_unicode
Except as soon as you tell the spammer what is blocked and what isn't, they
just add that to the playbook.

I agree with you that some measure of notification that there's spam filtering
would be good, but it's Google we're talking about. There's spam filtering.

~~~
markdown
> I agree with you that some measure of notification that there's spam
> filtering would be good, but it's Google we're talking about. There's spam
> filtering.

Nonsense! I had no idea this was a feature of GAE. I pay for the use of GAE,
and so long as I send less than my quota of emails, it's not Google's business
what the content of those emails are.

There's no reason that there should be silent spam filtering on outgoing mail
sent by my app. If they suspected that my app was sending spam, they should
have notified me and disabled my app. That would have alerted me to the
problem.

I find it absolutely abhorrent that Goole would just send private
communications sent to me by prospective clients into the trash where they can
never be retrieved or restored, and to do so without telling me is beyond
belief.

~~~
awinter-py
Nope, I don't think there's an outgoing spam filter. My _guess_ is that
individual messages are only stopped when they enter a gmail account; if
there's an abuse filter for email senders, it's probably on the whole account
after a manual complaint process.

The point: G is accepting cash from one group of people to send a ton of
messages (mostly unwanted by addressees) and earning ad dollars from another
group to stop unwanted messages.

30 years ago this would be textbook conflict of interest but the 'veil of
automation' means we're blind to a lot of unsavory business practices.

~~~
markdown
> My guess is that individual messages are only stopped when they enter a
> gmail account

That's been tested for and eliminated. My app logs that an email has created
and is going to be sent. The send_email function is called to send the email.
Then my app logs that the email has been sent.

The emails are sent to addresses hosted at both gmail and outlook. They are
not received. They do not appear in spam bins.

GAE has a bounce api that will log if an email is bounced back. In my case, no
reports are logged by my bounce handler.

The content of my emails has not changed for 2 years. For 2 years both gmail
and outlook received my emails in their inboxes (not spam bins).

Given all this, I'm pretty sure there _is_ an outgoing spam filter.

------
mikecb
Sendgrid is a great alternative and not hard to set up.

~~~
SuperKlaus
Also: mailgun

~~~
codenhagen
Also: Mandrill.

~~~
haimez
Not anymore.

------
daveguy
At least a few reported that it was bouncing at the destination once they set
up an appropriate handler. They are only "not being sent" when there is a url
in the message and that is a big red flag on the receiving side (not much text
but a url). It doesn't look like an App Engine problem. It looks like messages
being identified as spam problem.

~~~
markdown
Not the case for me. No bounces being reported. In any case, gmail and outlook
don't bounce messages for spam; They put them in the spam bin.

Nothing whatsoever changed in my code. I haven't touched it in a year, and all
of sudden email stopped on the 1st of April.

There's nothing in the changelogs to indicate there was a change that should
cause this behaviour. It's absolutely an App Engine problem.

[https://cloud.google.com/appengine/docs/python/release-
notes](https://cloud.google.com/appengine/docs/python/release-notes)

~~~
djsumdog
No, gmail constantly drops messages without putting them in spam. That's
actually a huge problem. Microsoft does the same. That's for e-mails that
don't even contain any links!

------
hartator
This is a documented API endpoint:
[https://cloud.google.com/appengine/docs/python/mail/](https://cloud.google.com/appengine/docs/python/mail/)
with quota.

------
mattbillenstein
Don't use appengine for anything -- ever. It's a terrible platform/product and
google would rather you move on to GCE anyway...

~~~
justinblat
Greetings folks! One of the App Engine PMs here. That couldn't be further from
the truth - we absolutely want people to use App Engine. If you want to manage
your own infrastructure, Google Compute Engine is great. If you want something
that scales automatically, and requires less management - App Engine is a
great option. We've actually made a ton of changes recently, specifically on
our support for docker based runtimes, new docs, and new runtime support (like
nodejs, ruby, python3, etc).

[https://cloud.google.com/nodejs](https://cloud.google.com/nodejs)
[https://cloud.google.com/ruby](https://cloud.google.com/ruby)
[https://cloud.google.com/python](https://cloud.google.com/python) .... etc

If you have any questions at all about the service, feel free to ask me
anything.

~~~
breakingcups
When will App Engine finally upgrade Go to 1.6? It's currently stuck at 1.4.

Also, why the long delay? Is it because of the switch from C to Go in 1.5?

~~~
puzz
Hm, I tried now with:

l.Infof("version=%s", runtime.Version())

...and it gives me:

version=go1.6 (appengine-1.9.36)

------
markdown
No change at
[https://status.cloud.google.com/](https://status.cloud.google.com/)

------
samskeller
How could you loudly stop sending email?

------
tedmiston
> I've set this thread to not be visible by anyone but yourself and Google.

Or not.

~~~
Klathmon
a few comments later he asked for it to be made public again.

------
Animats
This is what happens when you buy a service from an organization oriented
toward ad-supported services. The mindset is that it doesn't matter if the
user isn't being served.

