
Google Is Eating Our Mail - saintamh
https://www.tablix.org/~avian/blog/archives/2019/04/google_is_eating_our_mail/
======
reaperducer
Google has no incentive to fix these kinds of problems.

It's big enough that when someone complains that a message sent wasn't
received, the intended recipient will say, "I never have problems with my
Gmail account. It must be you." And the sender has to switch to Gmail to
reliably communicate with the outside world.

I wish this was just paranoia, but we've seen multiple discussions on HN about
Google programs and policies that alter the internet in ways that only benefit
Big G. It's like we're heading back to the days when people didn't know the
difference between AOL and "the internet."

~~~
prlambert
I'm one of the PMs for Gmail and hang around HN quite a bit. This is my
personal take, not an official reply.

It's simply not true we have no incentive to fix this. Here are a few:

Firstly, Gmail's success is entirely predicated on the health of the global
email ecosystem. Gmail does not, inherently, have any network effects (unlike
FB, messengers, any other comms tool of Gmail's scale). Email itself, of
course has a huge network effect, and _that_ is because you can email anyone
in the world, regardless of what email system they use. It's because email is
open. If we lose an open, healthy ecosystem with many providers, we'll destroy
the base we stand on.

Secondly, we care deeply about having positive relationships with developers
and all our users. I can tell you it definitely makes me sad to see articles
like this. There are going to be false positives, we will make mistakes, but
we certainly care a lot about fixing issues like this when we hear about them.

I agree Postmaster tools has been underinvested in and we could do much better
there.

~~~
dingus
I appreciate the response. From someone who has operated a mail server since
2002, it comes across that Gmail does not care about cooperating with small
but legitimate mail servers.

SPF, DKIM, reverse DNS, no blacklists, no open relay, longtime ownership of
IPs, etc etc. Using various mail testers returns a 10/10 deliverability score.

And yet, messages sent to Gmail always go into the spam folder, or are never
delivered at all. These are everyday regular messages, I have never used
mailing lists or sent bulk automated messages.

The issue is, there is no recourse, no fix, no acknowledgement of the problem
with false positives. There is no tool available to me to understand or
correct the "problem". Hint: this comes across as Gmail not giving a shit.

Gmail has a responsibility to be more accountable, even if these problems are
unintentional, because Gmail is such an enormous node in a federated network.

> If we lose an open, healthy ecosystem with many providers, we'll destroy the
> base we stand on.

Correct. Gmail is contributing to the erosion of email reliability. Please
course correct.

~~~
thaumasiotes
I've had gmail send the following to the spam folder:

\- Legitimate class action notices related to Amazon purchases.

\- Email coming from addresses to which I had already sent email. (!)

\- Email from my landlord.

\- Email coming from Google itself.

Based on the contents of my spam folder, which I have to check fairly often
because of the extreme overaggressiveness, I would be vastly better off if
nothing ever got filtered at all. [1]

>> There are going to be false positives, we will make mistakes, but we
certainly care a lot about fixing issues like this when we hear about them.

This doesn't sound honest, or at least not complete. People have been
complaining about this for years. I have personally been complaining about
this for years. The loss of obviously legitimate email is completely
outrageous.

It doesn't look intentional (look at that fourth category!), but it certainly
doesn't look like anyone is trying to address the problem.

[1] Yes, if spam filtering was disabled, more spam might get sent.

~~~
cpeterso
Your comment motivated me to check my Gmail spam folder and 10% (3 out of 30)
were false positives, two of which were pretty important. :(

~~~
wingworks
I've recently started checking quite regularly my spam folder as I've noticed
more and more legit emails ending up there. One of them being support emails
from TradeMe (one of New Zealand's biggest sites), keep getting put into spam,
even after multiple "mark as not spam", along with some other kinda important
emails from TradeMe. I've had to put in manual filters to force an email from
TM to skip the spam.

~~~
PenguinCoder
And yet, in my own none Gmail hosted email (fastmail), I currently have 2/55
false positive spam emails. I very rarely ever check it. To note I usually get
one actual spam (non newsletter blog spam) to my actual inbox, a month.

------
maxaf
> I'm sure there is no malicious intent behind this

How can anyone be sure of this? This is only one of Google’s practices that
seems to follow a pervasive pattern of eroding open internet standards while
presenting Google’s own proprietary implementation as somehow superior.
Eventually, the open standard loses all meaning because the most popular
implementation does not actually adhere to it. Meanwhile, Google reaps
enormous benefits in the form of additional signals for its advertising
business. How can this not be grounded in malicious intent?

> and that there are some very smart people working on spam prevention at
> Google.

There are some very smart people working on advertising at Google. The rush to
forget the primary nature of Google - it’s an adtech firm - is why they’ve
been allowed to skate for so long. Gmail’s spam filtering is just a pretext
for passing all email through a machine learning system. Sure, one possible
signal emitted by that system is whether a message is spam or not. Perhaps
this determination is conflated with wether the message is useful for ad
targeting: after all, when viewed from Google’s own perspective certain e-mail
messages contain no information which can be used for ad targeting, so they
must be spam. The user’s interests are clearly secondary to this.

So, back to the “smart people” working on this: at what point do we begin
judging engineers for working at Google? There’s a lot of highly vitriolic
criticism that emanates from Google’s workforce on a variety of subjects, but
how many of them would actually pull the pin and leave their employer? I don’t
have any statistics to offer, but it seems to me that we still have a ways to
go before Google has become completely drained of engineering mindshare.

~~~
joshuamorton
Email in Gmail isn't used for ad targeting, so this line of thinking doesn't
follow well.

~~~
calibas
Up until recently it was used for ad targeting.

------
jbergstroem
I'm also one of those oldfarts that insists on hosting my private emails for
me and a few friends. I've owned the ip since early 2000's and have had zero
"spam"/blacklist incidents. One trend I've been noticing is that
Microsoft/Outlook have regularly started to block me. I have to contact their
abuse which takes a day or so to get unblocked. To date, I've done so >10
times:

    
    
       hotmail-com.olc.protection.outlook.com[104.47.1.33] said: 550 5.7.1
       Unfortunately, messages from [<redacted>] weren't sent. Please contact
       your Internet service provider since part of their network is on our block
       list (S3150). You can also refer your provider to
       http://mail.live.com/mail/troubleshooting.aspx#errors.
       [VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
       FROM command)
       Reporting-MTA: dns; <redacted>
       X-Postfix-Queue-ID: 14A41FEB66
       X-Postfix-Sender: rfc822; <redacted>
       Arrival-Date: Thu,  14 Mar 2019 14:07:42 +0200 (CEST)

~~~
_dark_matter_
At >10 times I'd automate that process.

~~~
jbergstroem
Form has captcha: [https://support.microsoft.com/en-
us/supportrequestform/8ad56...](https://support.microsoft.com/en-
us/supportrequestform/8ad563e3-288e-2a61-8122-3ba03d6b8d75)

~~~
ApolloFortyNine
Is it illegal to pay for one of the captcha solvers? Ones like this are the
easiest, captcha solves cost something like 4 cents each, and all you do is
send the image to an API, which responds with the solved captcha.

------
i_dont_know_
As someone who runs their own personal mail server, this annoys me to no end.

I set up DKIM, SPF, and reverse-DNS records and resented every moment of it.
Even after all that, there's some chance that an email from my server will be
marked as insecure/spam or otherwise just not be delivered because Google has
come up with some new brilliant mail security/auth/permission scheme that the
world has to adopt tomorrow or be cut off from all Gmail users.

~~~
macspoofing
>Even after all that, there's some chance that an email from my server will be
marked as insecure/spam

Don't I know it. This happened to us but not with Google. We use Exchange
Online (Office365) and have DKIM, SPF headers configured. At some point our
emails stopped being received by bell and cox emails addresses ...
sporadically but sometimes for days at a time. It turned out that the spam
filter provider (used by bell and maybe cox) correlated one innocuous phrase
in our confidentiality notice (which is appended to all outgoing email
signatures) with spam and therefore marked the entire email as spam as well.

~~~
Leace
To be honest I don't like this confidentiality notes and from what I've read
(not a lawyer) they don't have any practical effect either.

~~~
reaperducer
I always giggle when I see those.

Especially the part about how "if you are not the intended recipient, you are
ordered to destroy all copies of this message."

It's my computer. You sent the bad message. You're in no position to "order"
me to do anything.

I suspect they're mandated by self-important Lower Middle Manager of the Year
Award nominees, not actual legal departments. The sort of people who never do
anything but push papers around, yet put a bunch of meaningless initials after
their name in e-mail signatures because they went to a conference.

~~~
Wowfunhappy
> Especially the part about how "if you are not the intended recipient, you
> are ordered to destroy all copies of this message." It's my computer. You
> sent the bad message. You're in no position to "order" me to do anything.

Best case scenario, someone listens and destroys the message.

Worst case scenario, the company is no worse off than if they hadn't included
the disclaimer.

Makes sense from my perspective.

------
jasode
_> I've been running a small SMTP and IMAP mail server for [...] around 15
years [...] I have SPF records and DKIM message signing setup on the domains I
use. The server is hosted on commercial static IP space (with the very same IP
it first went on-line) and I've made sure with the ISP that correct reverse
DNS records are in place. [...] Being a good administrator and a well-behaved
player on the network is no longer enough [...] Now every time I write a mail
I wonder whether Google's AI will let it through or not. [...] So far Google
has blocked personal messages to friends and family in multiple languages, as
well as business mail. I stopped guessing what text their algorithms deem
suspicious._

Yep, I know the author's frustration very well. I made a previous comment[0]
trying to warn others of personal email servers' _outgoing email_ being spam-
holed -- and yet some of the replies still argued I was overstating the
difficulties.

Everybody's risk tolerance is different. Personally, I just don't have the
bandwidth to administer my own private email server and constantly worry if
recipients are receiving my emails.

[0]
[https://news.ycombinator.com/item?id=15526089](https://news.ycombinator.com/item?id=15526089)

------
reaperducer
The last time a company screwed with e-mail at scale was Earthlink.

Remember for a couple of years, every time you'd send a message to someone at
Earthlink, you'd get an automated rely demanding that you verify yourself
before the message would go through?

Now I can't remember the last time I saw an @earthlink.* e-mail address.

Google has apparently learned from that and put the "error" in the 550
messages, where they can't be seen by the end user, and lead to non-helpful
resolutions for sysadmins.

The result is that the blame for missing messages goes to the sender, not to
the recipient's email service.

If Gmail at least notified the sender that there was a problem, then a pattern
of responsibility could be established. But this is just another dark pattern.

~~~
AJ007
If enough expected messages don’t show up, users are going to start thinking
gmail is broken. The problem for Google is that they are not the only trusted
brand consumers interact with. I’ve seen a lot of stuff start ending up in
spam that shouldn’t be there. If enough password resets, customer service
responses, and order confirmations aren’t delivered, it looks like Google has
a problem. If it doesn’t even make it to the spam box it makes Google look
really bad.

Google has definitely been changing things, everyone presumes it’s some rogue
ML system. My company had solid Gmail delivery for almost 10 years, and at a
fairly large volume. Now we can’t get it delivered any more. Very close to
showing users a message not to use Gmail.

~~~
luckylion
> If enough expected messages don’t show up, users are going to start thinking
> gmail is broken.

I doubt that. They will rather consider that the people (or their IT) that
send them emails made a mistake. It's Google after all.

~~~
bluGill
If several different IT departments all start sending that message though,
people start to think maybe google has lost it...

~~~
reaperducer
_If several different IT departments all start sending that message though,
people start to think maybe google has lost it..._

Many IT departments still build their networks tailored to Windows machines,
to the exclusion of others. Inertia is hard to overcome.

------
LeonM
I run a mail monitoring service [0], and we hear this complaint every now and
then.

A couple of things that regularly seems to trigger false positives in spam
algorithms:

\- no or misconfigured SPF and/or DKIM

\- no or misconfigured reverse-DNS

\- automatically included footer texts (confidentiality, copyright, safe a
tree don't print, etc)

\- regular automatic replies from the domain (such as out of office
notifications)

\- the use of embedded images (logos, human signatures, etc)

We sometimes joke that these triggers were built in by the algorithm
developers as a means of punishing those who litter their email with pointless
texts and images.

[0] [https://www.mailhardener.com](https://www.mailhardener.com)

~~~
throwaway2048
You can satisfy all of this, have an IP with an absolutely spotless reputation
that you have personally controlled for years that has never sent spam, and
still be arbitrarily shitlisted with zero recourse whatsoever.

~~~
LeonM
IP addresses are really insignificant in spam detection because spammers can
easily switch to a new IP address.

Like with Google's search algorithm, the spam detection algorithms are secret
and evolve constantly. So I'm not claiming I know how the algorithm works, but
it wouldn't surprise me at all if IP addresses aren't even used anymore in
Google's spam detection.

So no, you are not 'shitlisted' as you called it. There are no lists, there
are only algorithms.

~~~
throwaway2048
Whatever you care to call it is irrelevant, you jump through all the hoops,
and your mail dosen't get delivered.

------
tambourine_man
Same here. I love email for a variety of reasons, one of them for being an
open standard and protocol.

So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way.
And all is fine until you start receiving random hard bounces with no real
debugable answer for Google.

It got me deeply sad and questioning my decisions: since you can’t really
ignore Gmail, email isn’t in practice “open” anymore. So I might as well sign
up for Facebook, WhatsApp and the likes. It’s been years and I haven’t yet,
but it’s getting harder and harder.

------
megous
The real issue here is that if you tell some gmail user to contact their
e-mail provider to resolve delivery issues on their side, like this, they'll
just stare at you, not knowing what you're even talking about.

It's like there's an assumption that gmail is perfect, and the problem is with
the sender. Even if that was true, a normal mail hosting company would at
least tell its customer why the mail is not being delivered, so that the
customer can tell the sender what to fix.

The gmail recipient is never exposed to this side of google. So they don't
know what a nightmare comapny it is to communicate with.

Why should everyone and their dog be solving gmail users's problems with
receiving messages? It's such a demented system. It should be the other way
round. Recipients, via their provider should be solving their issues with spam
filtering and blocking.

If the gmail user would be blocked by my mail server, I would not tell them to
go guess what's wrong, fix gmail, and to have fun. It should not be acceptable
the other way round either.

------
john37386
I'm in the same situation as many people here. Started as a small personal
email projects many years ago just to learn. I really enjoyed it, reading all
the specs and making it always better year after year. Dmarc felt like a total
achievement.

I started to host emails to many friends, small businesses and even a SaaS I
developed. The subscription needs an email validation and I'm aware that the
activation email ends up in the Spam folder for the new customers using Google
emails. This activation email has everything from dkim, spf, dmarc, to
unsubscribe link, full physical address of the business, etc and still I can't
hit a good enough score.

I was thinking to start using Google service to send the activation link and
hosting my personal domains, but seeing that I am not alone, I will continue
to improve my little email projects.

Thanks all for cheering me up on this. I'm sure we can come up with a solution
and I would be happy to help. When do we start?

------
alyandon
I have logwatch logs that I email to my gmail address via Google's own SMTP
servers using my own Google credentials for authentication.

Hilariously, Google will flag these emails sent to myself using my own
credentials and their infrastructure as spam. I have no faith in them ever
getting this right.

------
bartimus
Google seems to not agree DKIM is setup properly?

[https://toolbox.googleapps.com/apps/checkmx/check?domain=tab...](https://toolbox.googleapps.com/apps/checkmx/check?domain=tablix.org)

~~~
avian
Thanks for the link. I think the tool you linked specifically checks whether
DKIM is set to allow Google's servers to send mail on the behalf of your
domain.

These checkers for example don't find any errors with the DNS record:

[https://dmarcian.com/dkim-
inspector/?domain=tablix.org&selec...](https://dmarcian.com/dkim-
inspector/?domain=tablix.org&selector=alpha)

[https://mxtoolbox.com/SuperTool.aspx?action=dkim%3atablix.or...](https://mxtoolbox.com/SuperTool.aspx?action=dkim%3atablix.org%3aalpha&run=toolpage)

One thing that this second one does find is the lack of a version number in
the record. I see that the RFC [1] got updated since the last time I checked
and having that is now recommended instead of optional. I'll add that. Thanks!

EDIT: after adding version record, Google's tool now shows green checkmark for
DKIM as well.

In any case, it's impossible to test a DKIM setup just by looking at the DNS
records. The true test is to verify the actual signature in the message. Last
time I checked, the mail on the receiving end (Gmail), did have "dkim=pass" in
the Authentication-Results headers.

[1]:
[https://tools.ietf.org/html/rfc6376#section-3.6.1](https://tools.ietf.org/html/rfc6376#section-3.6.1)

~~~
tssva
You may also want to add a DMARC policy. A dmarc record generator can be found
at
[https://mxtoolbox.com/DMARCRecordGenerator.aspx](https://mxtoolbox.com/DMARCRecordGenerator.aspx)

------
hazeii
Recently Google has bounced _replies_ to emails from a gmail.com address to
us, with a message about it being "suspected unsolicted email" (for a
reply???). That's just egregious (we're on a dedicated server at a reputable
supplier, same IP for years, proper email processess, not on any blacklist and
only low-volume usage - certainly no spam).

~~~
detritus
Yup, I literally lost jobs because of responses to enquiries hitting GMail's
spam trap, before finally giving up and moving from self-hosting to Fastmail.

------
verisimilitudes
This also mirrors my experience, as I host my own email and it's solely for
me. At one point, I couldn't successfully send any email to gmail, but then it
changed for some reason and simply gets sent to spam folders instead. Perhaps
it's entirely related to some people emailing me first that it gets through at
all.

I don't send much mail to gmail, though. Sans that, my only issue has been a
mail server that uses Reverse DNS, which I don't have set up, and entirely
ignores my email without it.

I suppose I can understand this if some people get a great deal of spam, but
requiring so much of this on an unencrypted message seems more like useless
reassurances than anything. I'm not criticizing email for being unencrypted,
but this seems more like another hoop to jump through than anything.

Also of note, almost all of the spam I receive is from gmail addresses and I
wouldn't be surprised if the invalid addresses that send messages demanding
bitcoin are also from gmail, but with fake From fields.

------
npsimons
I'll just chip in with everyone else who is also self-hosting:

I'm in a similar boat; been running email servers since before GMail existed.
My personal one I've been running out of a home server closet since 2001. I've
also done everything I can to guarantee I'm not running an open relay and not
sending email unsolicited. Have been mostly lucky so far, but occasionally I
will have people on mailing lists I manage (people I have met IRL and put them
on the list to organize group meetings IRL) not get email. Used to be other
stupid mail providers (AOL comes to mind), but these days it appears to be
Google, sometimes.

I've had this domain nearly twenty years and run email on it for that same
amount of time. I'm not going to "just switch", especially to a privacy
invading ad-spewing "alternative" that doesn't give me as much control. Fix
your damn servers, Google.

~~~
antt
Google seems to have entered terminal senility. In my circle of friends
googles search engine is pretty much useless, they have killed all the useful
applications we used and now this.

I can't help but think that we are seeing the google transition to late 90s
microsoft now.

~~~
chezhead
What do you mean by "In my circle of friends googles search engine is pretty
much useless"

~~~
antt
I mean that artists are unironically using bing as a better alternative for
image searches, academics have gone back to tracking citations + libgen/scihub
to find information and programmers are using duck duck go to debug their
code.

This is a huge problem since these people used to be the core of googles
search user base.

------
sneak
I have run a private mailserver for over 20 years, and I have the same
problem. There's no telling what Google will let through. I have a second
account at a medium-sized email provider, basically paying them for access to
the email deliverability cartel.

~~~
rc_mob
We need a new message protocol

~~~
tqkxzugoaupvwqr
I fear this will lead to the same problem: If it’s interesting enough for
Google, they will offer a free service, gobble up a large number of users, at
which point they can hold the new protocol hostage, add their own extensions,
and everyone else must follow.

~~~
sneak
It’s called XMPP, and Google ran a public gateway federating for Google Talk.
Basically nobody used it, and what little use it did see was mostly third
parties sending spam to Google users over the gateway, so they shut it down.

Centralized systems seem to give users what they want. This is why iMessage,
FB Messenger, WhatsApp, Snapchat, and Instagram are huge, and email is dying
(and XMPP failed to get any real traction in the first place).

------
pettycashstash2
Add another perspective to this. Once I updated my phone, and google would not
accept new google Authenticator codes after set up. Locked out of 15 years of
email.

~~~
laurentl
I had a fright last time I changed phones. After restoring my backup onto the
new phone, I was about to wipe the old one; I did a final pass through my apps
just in case... and saw that Google Authenticator configuration did not carry
over to the new phone (which makes sens from a security perspective but is a
PITA to manage). I had to re-enroll my device with all the services on which I
use MFA.

After this episode I made damn sure I had recovery codes stored in a safe
place.

~~~
Silhouette
When a family member had a phone stolen not long ago, the only major
functionality on it that they couldn't quickly and securely disable was the
Google stuff.

The procedures for doing so seemed to be unnecessarily complicated and
difficult to find when starting, ironically, from a Google search on another
device.

Worse, the security policies seemed to be fundamentally flawed, because they
kept insisting on some form of authentication based on a trusted device _when
the purpose of the transaction was to notify them that the trusted device had
been stolen_.

There has been an unhealthy trend recently of assuming that everyone has a
mobile phone and that communications to that phone/number are a good method of
authentication, without adequate thought to what happens if the physical
device and/or the associated phone number are compromised, or to whether
protocols like SMS are really suitable for this sort of application. And some
of the really important things, like banks and government services and email
providers (which are in practice a gateway to everything else you do online)
are often among the worst offenders. I don't know what to do about this, but
certainly raising awareness of this kind of problem would be a good start.

~~~
drivebycomment
> Worse, the security policies seemed to be fundamentally flawed, because they
> kept insisting on some form of authentication based on a trusted device when
> the purpose of the transaction was to notify them that the trusted device
> had been stolen.

Are you suggesting any random person without any authentication proof to be
able to just sign people out of their devices ? That would be a broken
security.

~~~
Silhouette
I'm suggesting that a security policy should actually be practical. There are
any number of viable ways to handle this. Requiring someone to possess the
device they are reporting stolen is not one of them.

------
gmailvictim
Add me to the victim list. I've been running my own mail server for 13 years
and have to tell everyone I'm intending to email to check their spam as that's
probably where my mail will be. I send a handful of mails a week probably.
Almost always goes to the bin if it goes to gmail.

~~~
ponytech
Same here: I have to text my friends to tell them I sent them an email and
check their spam. Nonsense.

------
petercooper
Lest anyone think Gmail is less susceptible to problems because of the sheer
volume of mail they handle (so any problem gets a lot of 'eyes' on it), they
had an amusing and long-standing bug that meant any email containing a link to
any domain name starting with "0x" automatically went to spam.

You could literally have two long standing, legitimately used accounts send an
email to each other containing a link to a URL like
[http://0xANYTHINGHERE.com/](http://0xANYTHINGHERE.com/) and it would be
insta-spammed. I suspect it was a hard coded rule to avoid people using "long
IP" URLs to circumvent other filters.. except there are lots of legitimate 0x
domains that _aren 't_ long IPs.

It was fixed sometime in the past year but I got a lot of use out of it in
talks I've given about email deliverability over the years.

~~~
lupire
What is your favorite public 0x domain?

~~~
petercooper
[https://0x46.net/](https://0x46.net/)

------
jussij
I run a highly customised PHPBB forum.

Without these customisations that forum would be overrun with all sorts of
spam.

However, these customisations only stop spam postings but can't stop actual
registrations.

Based on the users that I see who are registering I see a great majority of
these spammers love using Gmail accounts.

So while it is good that Google Gmail is trying to fix these spam issues, from
where I stand Gmail users seem to be a big part of the spamming problem.

Spammers love Gmail only because they can easily create spamming e-mail
accounts.

------
syntheticnature
Google? Any large organization. I help a small non-profit run their site in my
free time, and Google's actually been the nicest about our oldschool
forwarders and Mailman lists. Despite being on no public blacklists, AT&T
domains drop us at the border, with the only appeal process being via email,
and I've never heard back. AOL and Yahoo? Flip a coin.

Of course, email forwarding turns out to suck, but we're just going to suck it
up and move to G Suite for organizational email addresses and let folks
forward from there. E-lists, OTOH, I haven't found a good integration to
automate membership in the organization vs. G-Suite; perhaps it's time to just
move to a forum.

~~~
judge2020
FYI Google offers free GSuite basic for nonprofits:

[https://www.google.com/nonprofits/offerings/apps-for-
nonprof...](https://www.google.com/nonprofits/offerings/apps-for-
nonprofits.html)

~~~
syntheticnature
Indeed, the free GSuite is the plan.

------
superpie
A while back I'd explored trying to own my data, especially with email, and
found that the efforts involved in hosting your own email server were
tantamount to a full-time job.

The amount of fighting you have to do to stay on everyone's whitelists is
absurd.

~~~
ocdtrekkie
There's a good middlespace here, where you pay someone for email service who
isn't an adtech company. Sure, you don't host it yourself, but someone who's
not selling you is and you're their actual customer. (I pay FastMail,
personally.)

~~~
Nasrudith
I think you are still being sold even if you pay. Nothing really stops that.
Even if they give away or even pay the bigger providers they are ultimately
selling a white list. Since if they just accepted whoever gave them money with
zero policing of abuse, even a warning and take down system for spam then
nobody would accept their whitelist as it eventually becomes a blacklist of
"the one spammers use" and their business model essentially becomes a scam.

------
feanaro
Reading this thread made me realize the magnitude of this problem. It seems
very sensible that small email server administrators should unite into a
single effort in order to better publicize this issue, raise awareness and
shame Google into compliance.

Google seems to be suffocating the internet bit by bit on all fronts and it
needs to be stopped.

------
alfiedotwtf
Imagine a phone carrier dropping your calls before the phone even rang on the
other end. This is how I feel most of the time when I email people.

Every new recipient I email, if I don't hear from them within 2 days, I have
to contact them out of band to ask them to check their spam folder. The
problem is usually Gmail's heavy filtering.

------
lucb1e
Can confirm, my server is also not accepted by google for years now when
mailing someone with a business account. I'm not going to bother fixing
Google's problems and support the behaviour, though, I'll just contact them
through other means and let them know their email provider is blocking mail
addressed to them.

At work we use sendgrid because of this. Have to trust a centralised third
party to send out api keys. It's frustrating.

~~~
jasode
_> Can confirm, my server is also not accepted by google for years now when
mailing someone with a business account. _

Hey, I remember you were one of the replies[1] in my previous thread that said
I was exaggerating the "send emails" issue. Maybe your difficulty with Gmail's
mystery filtering algorithm will warn others that depending on personal email
servers for reliable outgoing email is a non-trivial endeavor.

Perhaps personal email servers are not impossible to set up but they're also
not as easy to debug as some make it out to be.

[1]
[https://news.ycombinator.com/item?id=15526127](https://news.ycombinator.com/item?id=15526127)

~~~
lucb1e
Good memory! A few things:

\- My comment in this thread was mainly about gmail for businesses, hence the
contrast between "I'm not going to fix their problems and just tell the
recipient their setup is broken" and the previous "it's doable". Other mail
providers are usually no problem at all, and personal Google accounts are also
better.

\- For email to personal gmail accounts it seemed to be enough to send a few
mails and get a good reputation. Since then (and since my previous comment), I
think I've started ending up in spamboxes again, but the number of messages I
send to Google is very low. It's not outright rejected, though, since I would
notice that.

\- I still think this part of my previous message is very true: "I don't think
we should dissuade people from doing it, especially if the fact that more
people doing it means that it'll be easier next time because it'll be slightly
more common. Many of us are in tech and the field is a small subset of the
population. Even if it's a small amount of servers setup by us, that could
make a noticeable on those working at bigcorps who write the hostile
receivers."

> Maybe your difficulty with Gmail's mystery filtering algorithm will warn
> others that depending on personal email servers for reliable outgoing email
> is a non-trivial endeavor.

I hope not. For what it's worth, I'm still at it and have no plans to stop
hosting my own mail! I'm very happy not to share my data with a third party
(the ideological part of it), but on a practical level, I also don't have a
spam problem and I can be sure that email arrives. No message has ever not
reached me due to spam filtering. It's also much faster than, say, Runbox,
which we use at work and takes at least 30 seconds for any sign up email to
arrive (too short to get coffee, too long not to be slightly annoyed). As
another data point, my girlfriend switched from 1und1 to my mail server
because it has some conveniences, without most of the downsides (since I
manage it all for her). She hasn't had delivery issues so far.

As a related tidbit, I've also recently started blocking Google from my
website. They're being a dick on the internet (not to me, it's more of a
solidarity move, see [https://lucb1e.com/!130](https://lucb1e.com/!130)) and
like with email, I'm not dependent on Google so I am again in the rare
position to say "no, I'm not going to play your game" to Google. I'd rather
encourage others to do the same than to give up and treat both email and
search results as a walled garden.

~~~
jasode
_> I hope not. For what it's worth, I'm still at it and have no plans to stop
hosting my own mail!_

You misunderstand the point I'm trying to emphasize. I'm _not_ trying to
dissuade _you_ from running your own email. There are real benefits to
controlling your own email server and it's great you've configured something
that works for you.

I'm saying I disagree with how some proponents _communicate_ the drawbacks.
They discount (or are blissfully unaware of) the real difficulties of running
a personal email server for _critical outgoing emails_.

In the previous thread and this one, there are several of us with _decades_ of
email admin experience saying it is tricky to debug "sender reputation" while
the opposing side insists Gmail/MSOutlook spamholes are easy to fix and not a
big deal. Therefore, one of the sides misunderstands the realities. With these
conflicting accounts, readers contemplating running their own email server
will have to decide who is more accurately reporting the state of the email
ecosystem for personal servers.

The author (Tomaž Šolc) of the article discussed in this thread had ~15 years
experience administering his own email servers and tried to do all the
"correct" things as an upstanding citizen of the email ecosystem and yet _his
emails still suddenly got rejected by Gmail_. Readers will have to conclude if
he got bit by forces out of his control -- or \-- he's incompetent and doesn't
know what he's doing. (From my point of view, the author wasn't incompetent
and his frustrations with Gmail's filter is a common reality. Maybe this time,
he can update DMARC to fix Gmail rejections... for now. But eventually, a new
Google AI spam algorithm will mysteriously reject his server's emails again.)

~~~
lucb1e
Fair points. I don't have much to add to your comment, so I'll just say that I
was nodding along with every paragraph. Thanks for clarifying what you meant
to emphasize, I think I understand now!

------
crank190426
In my opinion the Gmail web interface should expose a way to generate
something that is similar to a password, whose delivery is then controlled via
the web portal.

This is similar to the +whatevertag trick that gmail pioneered for tagging,
however that can be removed by malicious parties (spammers) via a simple
regex. So Google have almost all of the infrastructure but should just add a
bit more to get the rest of the way there.

What I mean in specific :

1\. I want to sign up to and receive your newsletter (you[re Ted) but I don't
trust you yet. so I should navigate to gmail.com, click something like
"generate another inbox", leave it set it to "For now deliver this mail to my
inbox", add the description "for Ted's Possibly Spammy Newsletter", and then
click "generate". It should give me inbox3943578423@gmail.com - similar to a
phone number but a bit longer and personalized to one recipient - and then I
should give that to the recipient to use, in this case the possibly spammy
newsletter. It should always be delivered to my inbox, as I've set. Once one
of the spammers sells my email address (for example I start getting advance
payment scams) I'll be able to disable further spam from there by sending it
to the trash but also know that Ted's newsletter is the one that got
compromised or sold it. You can do this today by going through the steps of
registering a new gmail address and turning on forwarding, but it takes like
10 minutes to do so. it should be like 10 seconds.

This should be possible because people always have easy access to the gmail
web interface. There's no reason it can't be a bit more like a social network
where you confirm it from the web interface as well.

that's my idea anyway.

------
z3t4
I've also run a mail server for a long time. Yahoo and Hotmail are probably
the most annoying, Yahoo because you can't do anything about it and Hotmail
for blocking entire IP-ranges. I don't blame them though, preventing spam is
very hard. Spammers always have spf,dkim,dmarc,reverse dns,signing, etc
meanwhile many legitimate senders has none. I'ts a hard problem to solve. I've
tried things like increasing the response time so the sender have to sit and
wait, or used spam lists, but there are too many false positives. One easy
solution is to make it illegal to send spam. Since my country made sending
unsolicited e-mail an offense, spam decreased a lot. The second problem is
hacked servers. Unless you are running a mail server, always block port 25 in
the firewall. So at least the hackers wont be able to send spam on your
behalf. Many ISP's already block port 25.

------
js2
I had my personal domain hosted with Google years ago. One of the reasons I
moved away is that emails that I was sending to other Google Apps domains were
bouncing as spam when the destination address was an alias. The bounce came
from Google Groups, which is what Google used for handling aliases within Apps
domains.

e.g. [https://pastebin.com/u48DAaLP](https://pastebin.com/u48DAaLP)

That particular example was sent via SMTP, but I had the same problem when
sending via the Gmail web interface, and it occurred sending to at least three
different Google Apps domains.

After I moved my domain off Google Apps (I switched to Fastmail for a variety
of reasons, but that issue was the kicker), I was able to send to those same
addresses without issue. In fairness to Google, I was on the Google Apps free
tier at the time, so there was nowhere to go for support.

------
cosmin800
I've always thought that, I also used to operate my own email server and I've
encountered the same problem with other email providers, always the big ones,
yahoo (there was a time when yahoo was big in email, at least here), microsoft
and now yes, google. What is even worse is that sometimes google just drops
the email messages, replies with 250 ok, but the email never gets delivered to
inbox or even spam. Definetly the big 4 or 5 or whatever are taking over on
all aspects of our life. Actually if I think better about it, they have
already done so, there was an article here on hn about blocking google, amazon
and ms network blocks, nothing really works as expected without them.

------
tssva
Regarding the issue with mailing lists for open source projects this isn't
just an issue regarding Gmail. DKIM and DMARC cause no end of problems for
mailing lists and it effects mail delivery to all major providers.

Mailman, which most open source projects use for mailing lists, have developed
work arounds to address some of the issues. Unfortunately my experience is
that many projects run older versions that don't have these work arounds or if
running newer versions they have not been enabled. Most likely because no one
has revisited the configuration since initial deployment on an older version.
After all they didn't start the project to spend their time being mailing list
admins.

------
surgi
Am I the only one here experiencing that GMail does reasonably good job in
filtering out the spam, while I have to visit the Spam folder approximately
once a week, just to randomly correct a mail or 2 a month? (not a Google
employee)

~~~
techsupporter
That’s not the issue, though. The issue is that Gmail are too zealous in
blocking and wind up blocking legitimate email with no way for the sender to
debug what happened. From your perspective, nothing happened because how do
you prove you never got an email you never knew was sent to you?

The people having problems here are non-Gmail senders sending email to Gmail
recipients.

------
neilv
If ProtonMail would standardize a client protocol that other MUAs implement
(not their "bridge" kludge), I'd start encouraging people to move there.

I'm not interested in ProtonMail's encryption (and it's potentially a
liability, attracting aggressive state action). I'm mainly interested in their
apparent respect for the privacy of users' private communications. And also
hoping that ProtonMail has a bit more reliable delivery than GMail.

In any case, rising competition lifts all performance boats, or something like
that.

------
zahllos
I've also hosted my own email for several years now, on several domains. The
one thing I have that does not appear to be listed is a DMARC policy. As the
comments on the blog post suggest, configuring DMARC works. I have no problem
delivering to either Gmail or Outlook.

I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA
has a fairly strict SPF configuration - any email with an invalid spf result
is rejected. This can come about because a legitimate company has
misconfigured their spf records (happened twice in all the years I have
hosted, discussions via postmaster@ helped them configure their dns
correctly), but 99.999% of the time it is a spammer. What is worse is that
rejecting email for domains without any SPF records can still result in valid
email being lost, in 2019.

In this specific case, I don't think Google are "being evil". They're trying
to reduce spam in the email ecosystem and they're doing it by using standards
they themselves adhere to (Gmail send me reports of dmarc statistics each day
google domains receive email from my box).

On the other hand, I do of course support either self hosting, or using
another provider so as to ensure we do not end up with a Gmail monopoly. If I
did not self host, I would find another provider like (but may not) Fastmail,
Posteo etc (I would have to seriously review the options, which I haven't
done).

------
username223
I'm glad I'm not the only one seeing these problems. I send mail from a
personal server, and recently discovered that my messages to people @gmail
were frequently sent to spam or simply dropped. I briefly thought friends were
ghosting me until I figured it out. I set up SPF, DKIM, and DMARC, but it
seems like the only way to get reliable email these days is to use Gmail,
Hotmail, or Yahoo (at least for a little while...).

------
rayiner
As far as I can tell, Gmail isn’t actually “real email” anymore, and you
shouldn’t use it for anything important. It’s verging in being a walled
garden.

------
ameliozanchi
Does AI compares whom data to define which message is important or No? It
doesn't track any specific way on evolution of mentality when it goes to
define data correlated to humans usage. Like others products on market.
Therefore, if you apply this thought to AI I guess they just made with basics
conglomerated informations gathering. That CAN´T move a barrier when
satisfying humans usage. It is just how you confront which message is correct
to say is relevant or no. AI can be freely a way companies uses their needs on
market influence because they're , on a very simple association, doing by
themselves, then people get a notion of arbitrariety on those machines that
create discussion to the whole universe you see about robot revolution etc,
but not the case. I guess they try to hard to make then appear as human to be
soon accepted by costumers, its just the way it is, it is not the time to
require maximum levels of perfection on how easy they're to addapt as a user

------
mullingitover
Gmail can be pretty aggressive with spam filtering, but I'm very happy with
the bias toward flagging things as spam. Spam was such an infuriating problem,
and lots of marketers would happily push the line if they didn't live in
terror of being blacklisted by Google. If only Google could the legacy
telephone networks like they run Gmail.

~~~
sbov
Flagging as spam is different. The OP is talking about rejecting it before it
even gets to your inbox.

If gmail is improperly flagging a message I get as spam, I can create a filter
to never send it to spam. If they're rejecting it like the OP talks about, if
I want to stay with gmail I have no way to get that email.

------
blunte
The black box is more mysterious to me. I had a client for whom I managed a
G-Suite account with a custom domain name. The domain was not on a blacklist,
and things generally worked well.

One day a colleague and I discovered that he had not received some of my
emails (intra-domain - me@example.com to him@example.com).

This is all within the confines of Google. Google had flagged some messages as
spam, and by what determination I could not fathom. The content seemed
perfectly typical.

I have had really pleasant experiences with G-Suite human support, at least in
terms of the quality of interaction. But they could not answer why some intra-
domain emails were being flagged as spam. I have suspicions that it would take
a whole team of G engineers to maybe identify what bit of logic in their
systems (incorrectly) marked some of the emails as spam.

It seems the beast (automation) is just almost not under their control
anymore.

------
c3534l
Isn't charging people to use other carriers one of the big things that got
Bell broken up? Google seems to tread in a of very dangerous waters. I wonder
if they're just not aware of how close they are to being shutdown entirely,
simply because so far they've always withstanded most legal challenges.

------
wichert
There may be a workaround here: configure your own mailserver to relay
outgoing messages through a service with a known good reputation. For example
relaying via sparkpost or mandrill should be trivial to do. You might be able
to do that under a free developer account if you don't send a lot of email.

------
specialist
Thank you for sharing your experiences, insights.

#1 - Does anyone send test emails and measure delivery rates? As in send
yourself a bunch of emails and see what happens.

USPS and its major customers and vendors do this with physical mail. They
measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector
General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006
[https://www.uspsoig.gov/document/undeliverable-addressed-
mai...](https://www.uspsoig.gov/document/undeliverable-addressed-mail)

#2 - What is the responsibility, liability for email relays to treat everyone
equally? For comparison, a US retailer has to accept US currency, but can
(sometimes) turn away problematic clients. Is there anything like that for
electronic exchanges, transactions?

------
johnnyhead
If people stopped using G in such high percentage, then they would most likely
need to adapt back to a more open internet, instead of trying to wall it off
around their services.

Reminds me of jabber.ccc.de that stopped providing new accounts because they
felt they were ruining a federated system.

------
beams_of_light
This is the reason I sadly don't recommend to customers that they run their
own email servers, and instead outsource it to GSuite or O365. Being on the
hook when emails aren't delivered for one or more of a dozen reasons that you
can't control is no fun at all.

------
strimp099
"Only when something like this happens you realize just how impossible it is
to talk to someone on the modern internet without having Google somewhere in
the middle." This says it all and is likely become a strategic imperative for
Google along the way.

------
duxup
>I wonder. Google as a company is famously focused on machine learning through
automated analytics and bare minimum of human contact

It should be noted that ultimately these efforts to "learn" result in ML, AI
whatever pointed AT you, not working for you.

~~~
Nasrudith
Well of course - don't trust the client is a rule for a reason. I get the
spirit and how it can suck but it seems to be for a reason. Anything which
blindly works for "you" can be abused by other "you"s like spammers and other
bad actors.

~~~
duxup
I think the catch with AI and ML there is "reason" but it's just math or
something that happens because.

If it is "reasonable" (trying to find a definition here) can only be
determined by humans.... provided they care or understand it.

I always quote this one but I really like it:

“Once men turned their thinking over to machines in the hope that this would
set them free. But that only permitted other men with machines to enslave
them.”

― Frank Herbert, Dune

I would argue that it is clear that to some extent, Google does not care about
the end results.

------
JohnFen
Interesting...

Like the author, I've been running my own mailserver for over a decade and am
very conscientious about ensuring that no attackers use it as a spam relay.

While the vast majority of the people I exchange email with don't use GMail at
all, so it can take a while before I notice any issues with it, I did happen
to notice that GMail was rejecting my outgoing email a couple of weeks ago.

This week, I finally got around to trying to address the problem (it's not
high priority because having GMail reject my emails isn't really a huge deal).

...and I found that it is working again without my changing anything.
Weirdness abounds.

------
kazinator
Running your own e-mail server for receiving SMTP makes a lot of sense in many
ways. I've been doing it for close to a decade myself.

 _Sending_ SMTP yourself (directly, without an SMTP relay service) sets you up
for trouble.

------
ddebernardy
What boggles the mind is that they've a very string disincentive to fix it: I
had to move away from my privately run email to Gmail for that very reason,
and I'd gather I'm far from being alone.

------
tempestn
Better that what I've been seeing, which is Google just sending emails sent
from my private domain to spam, with no bounce back or notification of any
kind. This is despite the fact that I send the emails FROM gmail using their
'send as' feature. Nor does it seem to matter if I'm writing to a contact whom
I've exchanged many emails with in both directions for years. If I send from
my @gmail address they always go through. Send the same content from my domain
and it's a crapshoot.

Edit: And of course, I do have DKIM and SPF configured.

------
dangjc
I tried to set up a Discourse discussion forum for our neighborhood this year,
and got stuck on setting up an email server that wouldn’t get blocked
automatically by all major providers. Email is not a healthy ecosystem. You
have to pay to send from one of the big trusted providers or else you get
blocked just because your email ip address is untrusted by default. Yes
fighting spam is important, but it’s shocking how much email providers rely on
simple ip filters and trust levels rather than AI analysis of email content.

------
Felz
This article didn't mention whether or not their servers are attempting
delivery using TLS/StartTLS, which is a good thing to check. I think Google
penalizes email delivered insecurely.

------
mtw
In the article:

> I can't tell other people to go off Gmail

I disagree. There are reasons to switch off gmail. Not just Google eating mail
but also for privacy reasons. Google knows all about your banking, eCommerce
orders, your media subscriptions, health issues and many other dependencies.

A good alternative is protonmail. It is private, has a mobile app, is a free
but you can also pay to support the service. I also consider protonmail much
more secure than gmail.

~~~
craftyguy
Or, if you don't want to commit yourself to yet another walled garden, there
are quite a few other email providers that to choose from.

~~~
silversconfused
Please list a few. I need 3 personal inboxes, 2 home business inboxes (bills,
etc), and the ability to make new inboxes for stupid startup prototypes you
buy a domain for, hack on for a few days with dreams of huge sales, and then
forget about for a year until the domain expires. You know, normal hacker
stuff.

~~~
gsreenivas
You may want to check out Helm - [https://thehelm.com](https://thehelm.com)
(I'm a co-founder)

------
reedlaw
I'm quite happy with the ease of setting up and running a Mail-in-a-Box [1]
instance. After seeing this article I did an experiment and while my hosted
mail sent to a @gmail.com address did not bounce it went straight to the Spam
folder. At least I can add other users to my email and enjoy private
conversations.

1\. [https://mailinabox.email/](https://mailinabox.email/)

------
aloukissas
Yet one more example of this. See [1], [2].

Definitely Google has done something to mess up their spam filter algorithms
in the last year.

[1]
[https://news.ycombinator.com/item?id=19536465](https://news.ycombinator.com/item?id=19536465)
[2]
[https://news.ycombinator.com/item?id=19500357](https://news.ycombinator.com/item?id=19500357)

------
jboy55
If I recall my time at a main-sleaze email sender, some ISPs will send SMTP
retry requests on random incoming email to see if there is a functioning SMTP
server on the other side of the originator's email. Often times, spammers
won't care about the email that's being sent, so won't respond, or in cases
where they are hiding the origin, won't be able to respond.

------
mises
I don't know that this is purposeful in all cases. I've had issues on a mail
host where sending an email to a different box on the _same domain_ would
reject the message as spam, refusing to even deliver it. With respect to
google, however, the fact that they offer a service of their own that
_conveniently_ can avoid this issue is more than a little shady.

------
angry_octet
Its like the deep learning has identified a feature it can use to determine
spam: is this from the top 10,000 domains? No? It's spam.

Spam from small domains might be pretty high as a category, but of course we
don't want statistical judgements about categories to outweigh the merits of
the individual. Maybe Google's algorithms have been watching too much Fox
News.

------
no_wizard
I often see and hear about this issue a lot and I wonder how other email
companies cracked it.

I use FastMail for instance and never have this issue, but I know so many
people who gave up on running their own mail servers at the small enterprise
level because of stuff like this I often wonder how FastMail does not have
these issues but others do. Is it a headers thing I wonder?

~~~
lolc
Fastmail is big compared to the operator in this article. Fastmail is probably
big enough that some Gmail engineers will know how Fastmail's SMTP chatter
looks. If you don't have volume you don't get that.

------
cmsimike
Someone posted on reddit the other day about (seemingly) the exact same issue
[https://www.reddit.com/r/selfhosted/comments/bbrvlt/any_reco...](https://www.reddit.com/r/selfhosted/comments/bbrvlt/any_recommendations_for_a_reliable_smtp_service/)

------
zzo38computer
I use my own SMTP server for receiving, but for sending I use the ISP's server
(I think I have seem to have read somewhere that not only Google but some
other services as well, will not accept mail from dynamic IP addresses). I
have not had problems that I know of, so far. Please tell me of whatever
mistake I may have made of such thing.

------
SourceParts
Gmail is rejecting valid content as spam and it never hits my spam folder. If
I was not expecting an email as I was in this case, I would be totally
oblivious. Something is up with the spam filter recently. They might have
decrease the tolerance for spam and as a result a lot of legitimate traffic is
being hit in the cross fire.

------
jasonvorhe
That's the price you're paying for a) self hosting and b) relying on
decentralized protocols that don't have sensible abuse protections.

People hosting their own servers enabled wide spread abuse due to
misconfigurations. Because everyone could do it and because defaults were shit
for decades, stuff like open relays were common. People defaulted to the wrong
ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam
would have killed mail by now if it hadn't been for major players like Google,
GMX, Hotmail/Outlook/etc.

Back in the day, greylisting was commonly regarded as a best practice, leading
to the impression that email is unreliable and prone to latencies.

I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy
to have a spam-free inbox every day and if this is the price for that, I'm
sure about 1-2 billion people are willing to pay it.

I'm quite astounded that there have been no updates to mail protocols in the
last couple of years to at least mitigate the most common issues, but all I
see are band-aids that are complex to setup and horrible to debug in case of
issues.

------
Tharkun
I've had Google blackhole me several times. Each time without any apparent
reason. It's impossible to talk to a human being at Google who is able or
willing to help. I've had similar problems with MS, but at least I was able to
talk to a human being and get the situation resolved.

------
ilaksh
To me this is more evidence that we definitely need public decentralized
platforms that can replace Google and once we have them we need to eliminate
the company. Private companies should not be allowed to monopolize platforms
and control our lives.

------
derefnull
Thanks for writing this up, this mirrors my experience setting up a modern
email server for personal & business use.

I currently self-host for non-mission-critical email, use FastMail for
business, and continue to use google apps for personal/mission-critical.

------
pera
If you run your own email server check out Postmark's DMARC monitor, it's a
free service:

[http://dmarc.postmarkapp.com/](http://dmarc.postmarkapp.com/)

( _tablix.org_ doesn't have a DMARC record)

------
majestik
He’s been using the same IP address for his mail server for 15 years. It’s
possible his IP is on a subnet range that regularly gets blacklisted by Gmail
due to other actors on the subnet sending malicious email/spam to Gmail.

~~~
megous
Presumably, Google's smart enough to identify individual IPs in the range that
don't send spam. I have a real shitty neigborhood IP wise, having my mail
server on a $1 VPS. Public Outlook.com drops all my mail regardless of content
for example, citing spammy IP range. Yet gmail somehow learned after a while
to pass most of my messages through.

The real problem is lack of provided reasons for blocking, so people waste a
lot of time trying to figure it out by guessing and trying random shit.

------
vinay_ys
Does anyone here use encrypted and signed messages on a regular basis? How
does gmail handle that on either sending or receiving ends? Is delivery rate
better or worse if it is fully encrypted?

------
rootusrootus
Reminds me of Spamhaus. Are they still around? When I worked for an ISP we had
to deal with them regularly. They'd happily hold you hostage over a single
e-mail.

~~~
jrnichols
yes, they are, and they've been really really good. haven't seen any of the
behavior you refer to. ever. now SORBS on the other hand... doesn't really
exist anymore (fortunately) but spamhaus has been great.

especially when used as a filter in Postscreen.

------
GnarfGnarf
I use non-Google email with SpamArrest, gives me complete control over
whitelist/blacklist. Legitimate users always have a way of getting through.

------
afarah
Ignore it and let people be angry (at Google for not receiving their mail from
various places). Meanwhile move to a decent service like ProtonMail.

------
jayalpha
Self hosting email is a pain in the butt.

Try fastmail infomaniak.com (can also buy email for external domain) gandi.net
(email included in domain)

~~~
jayalpha
PS: From the comments of the link

"We run small hosting company, if any domain blocks our clients mails we block
theirs and problem gets solved magically." :-)

------
jdmoreira
I had exactly the same kind of problems when I tried to set up a new mail
server.

In the end I just gave up and started using mailgun as a relay.

~~~
silversconfused
Can you tell us a little more about your setup? I have a home email server,
incoming only because of ISP restrictions, and I would love to make it into a
real email server for my family and business.

~~~
jdmoreira
It's just smtp relay really, nothing fancy. I still run my own mail server but
I relay outbound over to mailgun

------
alanlovestea
I am in the process to detach Google products, somehow gmail is the only
product I cannot find a working alternative.

------
DoctorOetker
I wonder if originally this technically constitutes an illegal form of
interception of mail in some jurisdictions.

Perhaps there really would be a lot more spam without such filtering, but it
points to the actual problem being elsewhere. Perhaps we need some kind of
cheap and userriendly (uniform but decentralized) email court system, and fine
/ ban email accounts that misbehave?

------
stratosmacker
I have a similar problem and it sounds like some here do as well.

What can we do about Google's email monopoly?

------
ackbar03
I for one welcome our new overlords

------
richardriko
Does google have legal obligation to receive mails from 3rd party smtp servers
?

------
muppetman
I have a lot of success by having put my mailservers in the dnswl.org system.

------
chrisfinazzo
Can someone explain in simple terms why, in 2019, anyone is running their own
mail server? Unless you are the sysadmin, server guy/gal, whatever, this just
sounds like signing up for a world of hurt doing it yourself.

~~~
simmons
Contrary to popular opinion, I don't think the technical aspects of operating
a mail server are all that hard. Perhaps I'm underestimating the learning
curve since I've been doing it for 27 years. (If you're running your own mail
server, you _are_ the sysadmin.)

The frustration comes with scenarios such as the one outlined in this blog
post, where small mail server operators get bullied even though they are doing
everything right. I can completely understand not wanting to operate a mail
server due to this situation, or not having interest in leveling-up server
administration skills.

For those of us who do have expertise in running mail servers, it's a shame we
have to deal with these obstacles.

~~~
calvinmorrison
The biggest obstacle of course is dealing with deliverability problems. As
most posters have commented here, getting people to accept your mail, and
accepting mail from reputable sources isn't an easy problem to solve. At
FastMail, a considerable amount of time is spent ensuring customers don't need
to worry about deliverability problems or spammy inboxes.

To me, that's the biggest reason not to spend time running your own
infrastructure for mail. Getting people to accept your mail reliably, is much
harder than adjusting some postfix configuration.

------
rdlecler1
Ironically Google's spam always seems to go through...

------
vectorEQ
They want your users to use gmail, that's all.

------
peterwwillis
Cue the *"but it's so easy to host your own email! Mine has been working fine
for years!"

Until this happens. Which it does very often, for lots of reasons that are out
of your control.

------
StreamBright
One more reason to de-Googlify my life. AWS has WorkMail I think it is time to
try it out.

------
TheTruth1234
Fair play.

I like my emails with pesto sauce, tuna, and cheese ... wicked combination.

------
5874-4b22-a4e0
Op should sign up for gsuite.

------
zoom6628
Only 3 people I know using Gmail any longer. I stopped back in 2010 and
"didn't feel a thing".

~~~
mrweasel
How do you really know? G-Suite has become extremely popular with business in
the last few years.

~~~
megous
You'll see something like this in the logs:

... status=sent (250 2.0.0 OK 1556280611 c1###########x.57 - gsmtp)

even if the e-mail address doesn't end with @gmail.com

------
imhelpingu
Have also run into this. Google is the quintessential evil tech corporation.
We should all start avoiding their services when possible.

