> If you’re worried about mail deliverability issues, don’t be — it’s more or less a myth in $CURRENTYEAR. If you set up DKIM properly and unlist your IP address from the DNSBLs (a simple process), then your mails will get through.
Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received.
I've checked DKIM and the DNSBLs, and everything seems correct...is it just a matter of not having enough volume for email providers to trust me?
No, it doesn't match my experience at all. Microsoft, Apple and Google all tend to randomly drop messages from addresses that were perfectly good the day before without any change in the sending mail servers configuration. It is super annoying because it actually forces me to use one of those to send email and that actually rewards those companies for creating this problem in the first place. I'd rather deal with spam than random delivery issues without any knowledge about it.
Oh, and I get more and more false positives in gmails spam folder as well, and since they're deleted after 30 days I now have one more 'inbox' to monitor.
Yeah that's my experience as well. And it's impossible to find anyone to talk to at Google/MS/Apple about an email issue -- unless you send really large volumes, then suddenly all kinds of support and tools become available. IMO it's a deliberate attempt to get more people to switch to gmail. But maybe I'm a cynic.
The problem is that providing email server debugging services is very expensive because not many people really understand the whole SMTP stack these days. You can't find anyone who will talk to you at these firms because there's probably only about 20 people in the company who could genuinely help. Additionally the world is filled with people who would like to run a whole email server for a stream of personal mail who would get it wrong and demand debugging services. That's why they spend their time building automated tools.
The problem the article discusses about needing to send lots of mail to qualify for these tools is because the systems don't scale infinitely - data storage still costs these firms money - and botnets mean basically every single IP on the internet will try to abuse your service at some point or another. So tracking deliverability data without any thresholding means tracking it for every connected machine in the world, which is hugely expensive, all to satisfy the tiny minority of people who feel passionately that they should run their own email server. The only possible justification for that would be recruiting related, but they don't have challenges recruiting.
The problem is the old cars. Really, old cars have absolutely no business on todays roads, and if you want to park your old decrepit car in my brand spanking new garage you should upgrade your car, better still, buy one of mine. Of course I can't be assed to make sure that your old car can still park in our garages. Your track width is ridiculous and the amount of horsepower you have just doesn't cut it today. So keep your silly little museum piece that's all of 10 years old now and drive around in circles on your own lot.
If you have a few million of those or are part of our circle of buddies we just might cut you some slack. But otherwise, no matter that your vehicle performed fine just yesterday, it's up to us to change the rules at will and demand that you adapt.
You're ascribing bad faith where there is none. Spammers make it difficult to keep the email system alive at all, even between big firms. If you think it's easy, go set up your own webmail firm and show them how it's done.
Carjackers make it hard to keep the road system usable at all, even between large fleets of cars. If you think it is so easy, why don't you set up your own fleet management system and show them how it is done, you miserly driver.
Apple will silently drop an email (no bounce message) based on text content of attached pdf files. This happened at my day job, we found the page of the pdf that triggered it via divide and conquer.
You could pay for hosting from a smaller provider on your own domain. Sure it costs money, but you don't have to support those giants and you get to keep your own address.
I was reading it as they are self-hosting and are therefore having their emails dropped. While I have had similarly terrible experiences with self-hosted mail, I have found even smaller dedicated providers have absolutely no issues with getting mail delivered.
Deliverability issues for personal servers are emphatically not a myth. One reason I moved off self-hosting mail a couple of months ago, after 17 uninterrupted years of doing so, was because of perennial problems with deliverability - and Gmail, while bad, wasn't even the worst; some large providers (glaring daggers at you, Microsoft) make it effectively impossible even to understand why mail is getting blackholed, much less do anything about it.
(Fastmail, in case anyone is wondering. $50 a year, and there's nothing I want that they don't do.)
It absolutely does not. I tried setting up mail on my own server and by all accounts I did everything right. Mail-Tester and similar services found no issues, I had all the optional records including DKIM, SPF, etc. and yet I couldn't rely on my mail going through.
Microsoft (Outlook.com) was particularly bad: mails were silently dropped without notice to me or the recipient even when I was replying to a mail they sent. I tried adding my e-mail address on my own servers to the trusted senders list on a MS account of my own and even then not all mail would go through.
Gmail ocasionally put me in spam but at least always delivered my messages.
At that point I gave up since I consider email to be too essential to be flaky and paid for Migadu. I have been very happy with their service and appreciate their pricing strategy (charging for number of mails sent, not the number of domains or mailboxes added). That said, I still dislike paying for hosting that I had sufficient capacity for on my own servers.
Then you have very low chance to get in spam unless you are sending spam and get blacklist. Time to time an gmail mx server will reject you, rate limiting, just try again with another and you will be ok.
Source: I run https://hanami.run and have to deal with this a lot due to nature of an email forwarding service that people usually use as one-off email (anything@domain.com for their one-off service) so it attract a lot of spam which I have to filter and make sure it won't go to gmail/microsft
I would say gmail is the best among big provide: Microsoft/Apple(iclouds) are garbage mail server where they just blindly trust DNSBL and block you no matter what. Gmail was way better.
However, if you use IPv6, it's a bit harder to avoid being flag as spam. It's really random and I don't know why yet and I contacted them and now "Waiting for 2 weeks to get a response"
Out of curiosity, where do you send the emails from? (Are the IP addresses from AWS/some other large provider? Or something else?) I've heard that the IP address block has something to do with getting marked as spam.
I've checked everything on that list and it's all good, and I'm sending from IPv4. The really frustrating thing is that Google's own Postmaster Tools [0] doesn't want to tell me anything without "a sizable daily volume of email traffic (up to the order of hundreds)".
If it doesn't have to do with the origin IP's neighborhood, my only other guess is that I don't have enough email volume, which seems like a catch-22...
At the time I allocate an IP address, it's was listed on DNSBL.
I tried like 30times to get a good IP and gave up. I instead just allocate a new IP, wait a week. Started to send email to myself every 15minutes to a gmail address. Anytime it isn't showed up in Important, I go in to move it to important. If it showed up as spam, I marked it as not spam.
After 2-3 weeks doing that, The IP is no longer listed on any blacklist and I started to roll them out.
Don't worry too much about IP blacklist. They aren't super important to gmail. If an IP stopped sending spam, they will be removed from IP blacklist eventually. Give it a week.
> "a sizable daily volume of email traffic (up to the order of hundreds)".
I send about 8,000 emails per day and somehow that Postmaster Tools not reports anything at all. I would say
If you send me an email vinh@hanami.run I can tell you what you can do to avoid being flagged at spam. Happy to jump in a 1-1 chat https://calendly.com/vinh-hanami/15min to help you on that.
Even replying to gmail messages or forwarding an older address to gmail will randomly fail. I've had some pretty important messages (for instance: invoices) randomly blackholed while all other mail arrived fine. It is super frustrating to answer someone's request for an invoice and to see that invoice then disappear without any notice that it was spam trapped.
To gmail point of view, they don't care about forwarding or replying to an email someone send you at all. All they care seems is about the message follow standard best practice(SPF/DKIM/DMARC/PTR) and send from a good IP. Sometime your IP can be flagged as spam wrongly on DNSBL. But it's very easy to get remove from DNSBL. Just submit a removal request, explain your situation and it's usually remove within a few days.
Also, the reason while your important emails are marked as spam is depended on gmail mx server you connected to. I don't know the detail but I have seen gmail mx server rejected/spammed my email, but on subsequent try if I connect to different one(by resolving other MX record), the email went through just fine.
If you send me an email vinh@hanami.run I can try to spot if anything may randomly flagged your email.
Not at all my experience. I was sending paying customers emails with their license key and a link to our starting page for years and it was fine. Recently our emails randomly land in spam v even though:
-we have never sent one unsolicited email
-100% of those emails are opened and read (I know because customers are activating their licenses). It's hard to imagine anyone mark the email with license they paid several hundred dollars for as spam
-we get tons of engagement, we answer around 30-40 emails a day
Something is terribly wrong with Gmail spam filter recently. Some of the emails from my customers land in my spam folder as well. They more often than not don't contain any links or anything out of the ordinary. Just a simple question or two.
It started happening a few months ago. It's a huge pain. All Google support tells us is that their magical box they can't disclose thinks our emails are spam even though there is nothing suspicious about either our website, the link or the content of the email. They can't tell us what triggers the filter cause it will make the job of the spammers easier.
I am now left to play the game against spammers: they try to imitate legit emails, if they get close enough then my emails are similar to spam, I have to modify them in random way, then spammers catch up etc.
Yay for machine learning I guess. Doesn't work so well even you have an adversary trying to fool you.
Btw, we use Gmail and it's Gmail spam filter which is causing problems. Google analytics is able to guess our monthly revenue and number of licenses pretty well. We send exactly as many emails. We are paying customer of Google Suite. It's still not their problem that their magic box puts our emails in spam but it's all on us.
It does not. I used to work for a non-profit that ran our own SMTP server, and we had issues with our subscriber e-mails getting marked as spam, mainly for Office 365 users. We set up DKIM, DMARC, and SPF, but it still happened. We ended up having to move to using Google's SMTP servers just so our e-mails would go through.
As someone who runs a discussion e-list server for a small volunteer org, Gmail has not been an issue with a good config, generally. The other big providers are annoying, to varying degrees, often silently dropping email. ISP email tends to be a pain in the neck, putting one on a blacklist if you exchange email with two different addresses they provide service to within one hour, though at least they tend to be noisy about it.
> Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received.
I use a gmail address and all my friends's self hosted mail have always ended up in spam by default
It does match my experience. I have a personal mail server with every bell and whistle working and my emails arrive to gmail addresses. My IP address was used by a spammer in the past, and took me some time cleaning it up in a bunch of dnsbl lists. But it reliably works now.
I have run small servers something like 18 years, my own personal something like 15-16 years. I still don't have much trouble. Only issue I had few years ago was spamhaus suddenly listing my IP as residental, that was annoying few hours.
Well, for personal matters I don't write usually letters without needing feedback. When I get answer, then my previous mail is clearly arrived. My friend has forwarding to gmail address, so he can compare both ends. Over years there isn't been more issues than mentioned spamhaus incident.
Ok, so you actually haven't measured your deliverability in any way. Yes, you can probably get emails delivered to your friends who have already emailed you in the past. Spam filtering is more of an issue when you try to email someone who has not engaged with you. If you actually go out and run some tests, you will probably find out a good portion of your email is blackholed by Google and Microsoft.
How to you really test that? Any provider is autonomous, you really don't verify delivery. You have few technical options, like DSN or MDNs, but servers or clients don't need honor them.
I didn't mentioned clients, I had few of them too. Some used internal mailing list style things, group addresses forwarded some outside service, like hotmail or gmail. I didn't had any complaints from them. Now talking about personal experience, your assumption is wrong. I don't change mail only with my family or friends. When you write merchant or government agency there is even more initiative to notice missing answer.
You can test by sending email to a bunch of different mailboxes on different email providers, and then seeing which emails land. I've previously used GlockApps for this test and I've been happy with it, but you can probably find other similar services too. I'd be interested to know the results if you decide to test your deliverability.
Some feedback: you should add a PTR record (reverse DNS) for studer.dev
and mail.studer.dev.
I also noted that your DMARC record has p=quarantine, which depending on
the details of your outgoing setup, may cause your emails to be placed
in the spam folder if DMARC fails for any reason. I generally advise
against using DMARC with any policy other than `v=DMARC1; p=none` or
else you're liable to run into issues, it's not a very good standard.
Otherwise, your configuration looks good to me. Also try
https://www.mail-tester.com to see what they think.
In case anyone is curious: I checked and the PTR record for my mail server (168.62.161.121 -> mail.studer.dev) is valid. mail-tester.com also gives me a 9.9/10, but I just tried sending an email to a Gmail account and it went straight to spam :(
I suspect it's probably a volume thing? As in I'm not sending enough email for Gmail to make any judgements about my mail server, since I'm only an individual. Of course there's not really a great way to resolve that...
It really depends. In our case, Gmail rejects our IPv6, but accepts IPv4. The IPs are from university network and they don't have any history other than being used by us. SPF, DKIM and PTR records are in place.
> Our system has detected that this message is likely suspicious due to the very low reputation of sending domain. To best protect our users from spam, the message has been blocked. Please visit https://support.google.com/mail/answer/188131 for more information.
This message is extra confusing because mail from said "sending domain" is accepted by IPv4.
50% of spam come from a valid gmail account, as in, using gmail smtp server, come from gmail origin(.google.com reverse dns) have valid DKIM. The only way to filter out spam is look at the content and run some simple ML on it.
50% of spam come from non .com TLDs such as .cam .work from random servers around the world: China, Bulgari, Hungari, France, Germany...
But it's very easy to block other TLDs because their IP are on DNSBL already. But what can I do with gmail? Gmail IP get on DNSBL all the time.
Example: 209.85.222.196 is listed right now on spam.dnsbl.sorbs.net and ips.backscatterer.org
But at the same time, this is just the nature of an email service. If we have a large user base then no matter what we do, a small percentage of users are abuser which are successfully to send out spam using our services before we can detect and ban them.
On AWS SES, they requires user to keep a reputation of less than 0.1% emails flagged as spam. Go over than that and you will be put on a review queue. Reach 5% and get banned permanently.
But before that occurs, spam are send out already.
I'm confused. What, exactly, is the author looking for Google to do about these "spam registrations"?
The abuse problem that an identity provider can solve is bulk account creation, but a) they are already doing a better job of it than any other free identity provider, b) given how low the numbers in the post are, this is clearly not any kind of bulk account issue.
(Also, not stating either the total number of suspensions or the proportion of suspensions that are for gmail seems like a conspicuous omission. The author states that 41% of all registrations are from gmail, and the absolute highest proportion of spammy registrations is 63%.)
It is an issue of bulk registration. The reason why the numbers are low is because we detect it fairly quickly and teach their shape to our abuse prevention tools.
We don't keep stats on discarded registrations, and usually the spammer gives up or changes tactics when they realize what's going on. How exactly does this invalidate my thesis?
These are accounts that are abusing your website, right, not sending email spam. The problem is Google doesn't know the accounts are bad so its systems never learn to spot the tactics being used to get past the bulk signup controls.
Only slightly related but it seems Google has also become very bad at filtering spam too? I keep getting agency spam from Indian companies, all with very similar characteristics (HTML emails for IT jobs in the US, where I don't live, from Indian companies). I marked hundreds of them as spam, and even a basic bayesian filter would be able to easily filter this but somehow gmail let it all go through.
I used to never get spam in my GMail. Now, every once in a while, I get spam in the form of Google Drive spam (pharmacy stuff) and then I get "You've won Amazon Rewards!!" type of email. It's very strange.
I don't get any spam in my inbox, but I do get non-spam messages marked as spam all the time, to the point where I have to check my spam folder once or twice a day.
Anecdotally, my inbox is filled with spam from google forms. Somehow scammers are convincing google forms to send me an email with their message. I imagine it is difficult for my provider to distinguish spam google forms emails from non-spam.
All of these cases are after the user has verified their email address by acknowledging an email sent to their inbox, so we can be certain that these are actually from genuine gmail.com addresses.
So, gmail makes up 41% of registrations and also a large percentage of suspended accounts? This doesn't seem very surprising to me. Also, for me, just because an account was "suspended" doesn't mean it was spam.
I guess I'm just lucky. My current personal spam corpus goes back to 2019-05-30. Of the 39454 messages only 80 of them have a ".google.com" server as the immediately preceding "Received:" hop.
It is unclear if Gmail is proportionally any worse than the others. Is the 119 suspended accounts more or less than 41% of the total number of suspended accounts?
I get what you feel, but on the flip side, if Google was even more aggressive at policing accounts and even hit 1-2 false positives, only for a scathing post to follow on a popular orange news aggregator, how is that a better situation?
False positives are okay. Google gets scathing posts because they suck at customer service. Hire people to do customer service like everyone else does already!
How does customer service detect a false positive as such any more than a machine can, without 1) enabling social engineering 2) damaging the privacy of unrelated users by snooping in their account?
Customer service can offer a path of recourse for the false positive. If the AI understood how to deal with it it wouldn't have closed it and since nobody is there to explain the AI must have missed something you're just shit out of luck, birth certificate in hand your account was banned because you were underage.
Recently in the last 6 months Gmail has gotten significantly worse in filtering out spam, I'll get really strange messages showing up in my inbox, where the red ssl ({{domain}} did not encrypt this message) shows up. It's quite frustrating
> If you’re worried about mail deliverability issues, don’t be — it’s more or less a myth in $CURRENTYEAR. If you set up DKIM properly and unlist your IP address from the DNSBLs (a simple process), then your mails will get through.
Does this match most people's experience? Or is he just talking about the deliverability of smaller providers like Migadu? I run my own mail server and can never send emails to people on Gmail/Microsoft 365/etc unless it's in reply to an email that I received.
I've checked DKIM and the DNSBLs, and everything seems correct...is it just a matter of not having enough volume for email providers to trust me?