It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
> It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.
Yes, I have been bitten by this many times. Every time I see a topic like this one I try to help that person by offering technical advice, basically doing my consulting job for free for everyone to read. And many times the post gets downvoted till its dead.
Slowly I learned that people don't want a solution to their problem, they just want to vent their (understandable) frustration about $BIGCORP. But it still hurts to get downvoted when you just try to help.
People ignore the email spammers, SEO spammers, automated password crackers and blame google for the solutions they put in place to combat those things.
It's impossible to reason with them, honestly I think it's just best to ignore posts like this.
I have my mailserver configured in pretty much an optimal way, with all the special bells and whistles, have never sent spam, and never will.
Gmail refuses to deliver my mail, so please don't lump me in with malicious people and then say my concerns should just be ignored.
In the end, TFA is from a subdomain that seems to have a lot of spam, and to avoid it may mean using A commercial option, if not a specific commercial option. It sucks.. but the mail relays I mention above are relatively cheap and easy to properly setup with limited issues on the other side.
There is usually some misconfiguration somewhere that goes overlooked. Happens to the best of us :-)
What is your opinion (or others' if they'd like to chime in)? If you enjoy running your own, are there any guides in particular that you'd recommend? Does the future of email look even more prohibitively complex for self-hosting?
Following a guide to setup a basic email service is quite easy to do, but after a while the lack of proper implementation and maintenance of the standards will start to cause deliverability issues. Those are frustrating and most end up writing a long rant on why Google is evil because their email ended up in spam. When in fact, it usually is a misconfiguration, but those can be really hard to detect.
IMO there is still great satisfaction in running any internet service yourself. But, if you rely on this service to work, it just no longer makes financial sense to do so.
It takes resources to keep any internet service running correctly. Even the simple stuff can quickly get overlooked, I have seen so many self-hosted email servers that have expired certificates (like with the domain of the OP) or lack support for any of the modern security extensions such as DMARC or MTA-STS.
For most businesses my recommendation is not to run it yourself. Like you wouldn't run your own DNS service anymore. Of course it can be done, and in some cases it makes sense to run stuff yourself (usually when operating at a very large scale). But it almost always makes more financial sense to use a hosted email solution.
We haven't written a guide (yet) on setting up an email server with all the bells and whistles. But if you want to read more on all the email security standards, have a look here: https://www.mailhardener.com/kb/
For delivery, if you are really just using it for personal use a mailgun or sendgrid account can allow you to handle most delivery issues, while still handling inbound.
Which still leaves the other side, actually dealing with inbound mail issues which is just painful with either too much or too little filtering. The likes of google and MS have teams of brilliant people to handle dealing with spam, and despite some little guys caught in the middle, is still far better than I could ever do on my own.
In the end, if you want a career in IT, or find it interesting, I'd say go for it. I've often thought about building my own end to end open-source mail service aimed at ease of administration. If I was ever rich enough to not have to work, that's one of the things I'd probably do after a few months off.
I'd suggest starting with a secondary domain and using Mail-In-A-Box on a VPS or smaller Cloud host. It's a decent starting point, and there are many other options. One of my favorites is SmarterMail (commercial, windows only though). I also ran a BBS for a number of years using Synchronet, which does email/pop/smtop and even nntp for group messages. I did have it configured with SendGrid at the time for outbound for a while, which handled delivery issues for me.
You will have headaches if you take it seriously and/or use it as a primary service without using a delivery service like MailGun or SendGrid, and even then you probably will have other headaches. My point isn't really do discourage so much as let people have a more realistic understanding. The issues the OP has are real. However, there is so much junk from bad actors that the well is poison.
Please make your substantive points more thoughtfully.
What sort of config do you require from the OP? Their domain name is public and uncensored right in the top of the post. Their configuration has been cross-checked by many people so far. A lot of folks claimed that it has issues (like a `?all` in SPF, or `p=none` in DMARC, or some other nonsense that any spammer would easily get right), but those issues have been rebuked as not being significant to the issue at stake, plus, there's absolutely no confirmation that fixing these "issues" would resolve the problem, either.
Google provides a service to its users, they don't provide a service to you.
Unfortunately for you, the e-mail providers' customers (the users receiving the mail) get to decide what is an unacceptably high false positive/negative rate --- and not you. If e-mail recipients get too much SPAM or if they get mail that they want landing in their Spam folder too often, then they will switch. This provides the economic incentive for mail providers to Get Things Right.
Unfortunately, there is so much spam out there that if you are too relaxed with letting marginal e-mails (or e-mails from marginal network neighborhoods through) the false negative rate could result in so much spam in users' inbox that they will get mad, and switch.
However, it can happen that the email provider can be too sensitive, and there can be too high of a false positive rate. For example MIT recently tried relying Microsoft's spam filtering system. Way too much stuff landed in the spam filter, and while you could go to the spam filter and manually mark e-mails as "not spam" and eventually Microsoft's ML algorithms would hopefully figure it out, the false positive rate was too high and the MIT community (especially the ones with the real power, e.g., the professors) rose up with an outcry, and MIT abandoned using Microsoft's Spam filtering service.
They can be pile driving right next to the machine room, and causing disk drive errors, and Physical Plant won't care if the I/T folks complain. But if some Sloan School professors complained that it was disturbing their cogitations, it immediately stopped and Phys. Plant called a "stakeholder meeting" and it was really obvious who had the power.
All customers are equal, but some customers are more equal than others.
Even the particular people who irrationally hate something are obviously likely to also run across the good reasons for hating it.
It's not reasonable to say "criticism X doesn't count anymore because we've heard it too many times".
Pro tip: start by briefly acknowledging the frustration, then offer the technical solution/workaround.
People forget sometimes that the person on the other end of the screen isn’t a computer that is completely unmoved by silly things like emotion and frustration.
As a hobbit* , I'm actually pretty happy this whole conversation is starting to gain more traction.
My domain is always green on every test I could run (mailtester.com & all) yet some of my mails go to the junk without any warning. At least warn postmaster when you decide to block a domain because of bad neighbour/hosting!
* This specific reply of the chain is a must read and is where learned that I'm actually a mail hobbit. https://email@example.com/msg08958.html
This exactly. I've been a Gmail user since the day when they've required an invitation in order to signup, and I'm not quite a Google hater, either, but it's extremely disappointing to see the number of people who simply defend Google against the wrongdoing irrespective of whether or not the complainant has a valid complaint, like in this very case.
What happened to the whole notion of believing the victims? Why no matter how hard you try to explain your setup, and the things you've tried, and that all the automated tests pass, that you still get blamed to be the culprit for the issues you experience when dealing with someone who has a monopoly on the service at stake, instead of considering the possibility that maybe it's Google Mail having the monopoly on email that's an issue here.
Also, if you aren't sending a lot of mail, consider relaying outbound through mailgun or a similar service (I used to use sendgrid myself). Nearly zero issues when doing this on the lowest level paid account. That said, inbound spam was so painful to deal with I just gave it all up.
Princeses were the customres which sometimes required a lot of customer support, but paid really well because they purchased a lot of services/VM's. Hobbits were customers who basically minded their own business, and didn't guy a lot of stuff, but didn't cause any problems.
Lizardmen were customers which had possibly clueless marketing departments that would occasionally send e-mail marketing campaigns which crossed the line, but which also sent plenty of legitimate e-mails, and which were mostly trainable after the ISP/cloud provider smacked them on the nose with a rolled-up newspaper --- and Kobolds were spamvertising companies that did nothing but send spam, and were constantly switching ip addresses / VM's, and knew d*mn well that they were doing something evil, and were not trainable.
The problem is that if the ISP / cloud provider has the vast majority of their customers being Kobolds and Lizardmen, it becomes economically hard to just cut off the bad actors, because the vast majority of their customer base are bad actors. If the provider originally had a good mix of all of these customer mixes, but wasn't proactively cutting off the Kobolds and educating the Lizardmen, then their network block would start getting a bad reputation, and all of the major e-mail providers (not just GMail) would start blacklisting the netblock, or at the very least, treating any e-mail sent from that network block as a strong SPAM signal, and this would cause the other customer types to find other providers.
So if you happen to be a customer of such a network/cloud provider, you could try complaining to the provider, but they are really in between a rock and a hard place, because if you're the 1 good hobbit and all of the otheer customers for that provider are Kobolds and Lizardmen, what is the provider supposed to do? You're probably better off switching to a better provider. Unfortunately, the better provider may also cost more, but you get what you pay for.
Because it's really infuriating if someone writets to you, you put time to help them with something, and their demented (did I say it's infuriating? :D - also what kind of company can't detect something is a response, if you even quote the original email, it's ridiculous) e-mail provider trashes your response.
Sadly, I'd imagine some users don't really have a choice, e.g., an employee whose company is using Gmail from G Suite for its email, so, it'd be an uphill battle for sure.
So if someone can't be bothered to check so that I'm not wasting time responding to their trash folder, that's a good filter in the first place.
Anyway, better thing might be responding by adding a (F)AQ item publicly, and just sending a link.
The common thread among many of the bad-Google stories I've seen in the last 10 years is that Google has made a deliberate decision to not give two shits if they accidentally hurt a small minority of their users. People get messages out of the blue that essentially say "We've just wiped out your email account, we're not telling you why and there's no one you can appeal to." I don't know if this particular guy is on the level or not, but it's not credible that all of these people are secretly spammers and scammers. Occasionally you can get things fixed if you know someone at Google or your complaint goes viral, but that's your only option.
I'm sure someone at Google did some math and concluded that the ill will of a few customers costs less than maintaining a support department. That's not an excuse. I don't think we should expect Google to never make mistakes, but we should absolutely expect some avenue to correct those mistakes.
Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.
If you can get into people's email inbox, there's still quite high conversion rate - better than clicks/videos on a pure percentage basis. Even the author wants to reach out to people in a cold fashion.
As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.
As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on.
I can't have it both ways. I would not be on Gmail if it was like yahoo. My goodness... my parent's inbox, on yahoo, makes me anxious.
Google's spam models freaking suck donkey dick. I use Fastmail but my university uses Google for email. A service we use (Piazza, class Q&A) sends notifications via email. Over the years I have gotten probably about 1k email notifications from them, and replied to a few of them too. Google has recently started marking about 5% of their notifications as spam. 95% of the notifications come through; 5% get rejected. There's no discernible logic to this.
There are organizations and people within the university which have me on their mailing lists; with them too 90% of the emails get through but 5% get randomly marked as spam. And they're perfectly normal emails that Google just decides to move to spam. Even with people that I have had email conversations with, it just spams random stuff.
Every couple of months I check my Fastmail spam folder just for the lolz of seeing what's in there. With Google I have to check spam every day or I'll miss important stuff.
There are tons of people who have posted very similar experiences on HN. On this thread I just saw another person comment that Google's own emails go to spam - I've observed this too on my personal Gmail account which I occasionally check.
The spam filters have a tough job, but other companies are succeeding at keeping all the spam out without a gigantic false positive rate. Google's filter is hilariously bad.
I've seen this with my SaaS. We ONLY send transactional emails like purchase receipts and login links (using postmark). We're very intentional about not sending anything remotely resembling marketing or spam, and sending as little as possible. Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.
I've noticed that in the past year or so gmail has made hitting the spam button more explicit and now have a confirmation message. Obviously they did this because some people were simply mashing that spam button instead of the delete or archive button.
I don't even blame the users. Most don't know the impact it has. But bad actors sending tons of spam have trained them to be cynical of any generic form email.
Google does put some senders often into spam for me, but not always. I'm absolutely sure I've never reported them as spam, on the contrary: I've marked their emails as "Not Spam" when I caught them in the spam folder.
That had, so far, no influence on the filter. Those aren't "weird" emails either, they are in my language and individual communication, not mass mails.
Maybe gmail shouldn't have the spam button right next to delete :)
At least that's better than web Outlook's invisible, until you click on the sender, microscopic disposition buttons. I can't count the number of times I've deleted an email instead of opening it.
I have a relatively high profile email address (first initial and last name) and I almost daily get at least one form of "receipt" from a transaction someone made where they entered my email by mistake.
Sometimes I'm in a bad mood and I'll mark anything I see like that as spam partly because its frustrating and partly because I suspect a good number of these emails actually are spam masquerading as receipt/login/etc emails.
And I just don't have the energy to figure out which are legit or not.
And yet other companies with spam reporting features are able to maintain a lower false positive rate with their spam filters.
I've considered adding an opt-out checkbox during checkout but I honestly doubt it would ever be unchecked and if it was opt-in we'd have users contacting support asking for it because they didn't realize they had to check the box. Either way it's one more toggle on the UI which I'm not a fan of unless absolutely necessary.
I'm curious if you've run a business and experimented with not sending receipts, and if so how it played out?
The purchase history on website disappears, logins get forgotten and so on. Only receipts in the mailbox stay forever, as I want them to.
Also, a lot of users do perceive receipts as spam.
My own server has a good reputation, but it meant setting up SPF, DKIM, DMARC etc.... Where it was the most painful was with microsoft email (I don't know the name of their service now, still live maybe). By default mails goes to spam, but really quickly they goes in limbo if people don't respond (on the smtp side the email is accepted but the recipient never see it, not even in spam). On new microsoft accounts I was always able to send a few emails in spam before it went into limbo.
I ended up creating ~20 different accounts on their system (it was before they enforced giving a phone number) and had random discussions with myself for a while which was enough to build a reputation and not being put in spam anymore.
I don't know if this behaviour was intended or not, I try to get in touch with someone from microsoft but couldn't find a way. Since then I do strongly advise not to use their system because that makes it really unreliable.
I’ve got everything setup properly, all test websites come up green, and google receives my email just fine. SPF, DMARK, DKIM, PTR, HELO are all setup correctly. This isn’t an isolated problem either, the ms forums are filled with people reporting similar problems.
Slightly tangential ... I heard recently that "proper" in-line responses (ie, not top-posting) is somehow penalized by Microsoft (or Exchange online or Outlook online or ... ?) and was advised to stop my very proper 1990s style in-line email quoting/responding and just top-post to avoid either non-delivery or something like the body of the email not being shown ...
Or something ?
Does anyone know what I am referring to and can shed some light here ?
But yeah, that's usually more an issue with the mail client (or perhaps the mail server if it's being exceptionally stupid and modifying emails before final delivery). Top-replies became the norm for some god-forsaken reason, and now clients treat the quote and everything below it as entirely optional to display.
I found an important email (from 3 days ago) from a recruiter asking for my CV in it.
Yeah, Google's spam has erred too far towards making false positive errors.
As another data point, for me and friends I it beats competitors by a large margin.
I guess it's another case of survival bias. If it works you won't notice that it does not work in all cases.
Anti-trust regulators should specifically look at situations where Google began directly competing with other companies and those other companies gmail messages are being put in spam while Google’s messages are not. Whether or not it is intentional, it is a problem if Google is whitelisting their own stuff but not competitors.
Ultimately what is going on here is Google is using more deep learning for spam detection. Your friend’s email ending up in spam may be a false positive. It may also be a test, to see if you notice it.
As a bonus, if you require opt in, I won't be able to cancel your car reservations with budget ROBERT.
I absolutely click spam on every single one of those pieces of mail, and I'm glad that they are being penalized for it.
We run our own mail and DNS servers and, as you might imagine, have never sent a single HTML email or any email with any kind of "element" in it, etc.
Just plain, ASCII text.
We've got a 15 year clean IP, perfect score on the "mail config checkers" and almost every single email we send is in response to an email from that sender.
And yet ... often enough to make me anxious, we get marked as spam.
It would be good to have a central registry of mail forwarders so that SPAM is not detected against registered IPs and then issues can be followed up directly with anyone who abuses the system. Do you really want to greylist Amazon SES because one of its users sends out spam?
On top of that, of very old-school users on Mutt or Pine who prefer text over HTML, I'd think a significant portion of them don't have their mail pass through Gmail's filters.
You can't use part of Google service without being entangled in their new protocols, the way they code and build things, run operating systems or servers, browse the web, and so on. You have to run your business the Google way now. The most scary part of this - the monster is completely deaf. Cry as much as you like, it's not going to listen. Even if you bring thousands of gifts to it every day, you can be erased from the web in an instant, and it won't even notice.
It's not too evil (yet!) but it has so much power it's scary.
Interesting how different things are for different users. Gmails spam protection isn't working at all for me. Confirmation emails from reputable sources get into spam, while really obvious spam messages get into my inbox. I mean.. really, really obvious messages. With "dfdf" a thousand times at the end just to make the message longer. Or these porn-bitcoin-scam messages.
It's just interesting how perspectives differ on this type of thing.
When you refer to specific threads going back and forth this may be. But here's my counter example.
I sent emails to my girlfriend's gmail address from my own domain hosted on a mail server run by a reputable company for years with no problem, whatsoever.
Then, on vacation in Thailand, my rather elaborate trip reports for which there's really no plausible explanation that they could be spam, suddenly bounced.
Sent from the same domain from which I sent mail for years and Google suddenly branded me a spammer and refused to deliver those mails even to her spam folder.
The auto-reply from Google is about as useless as you would expect. There are a few links, which are neither informative nor do they help to resolve the issue.
I worked around the problem by sending mails from my own Gmail account, but it left a really rotten feeling around the lines:
NICE COMMUNICATION YOU HAVE GOING THERE, WOULD BE A SHAME IF SOMETHING HAPPENS TO IT!
So I guess your assertion is wrong. Our exchange went on for a minimum of four years. Always from the same account and suddenly I was deemed an evil spammer.
It really sucks.
or they tweak them to cause more people to switch to gmail.
That, actually, is my suspicion. It's easy to brand that as "protecting their user base", but in reality make it so cumbersome that you're not able to send mail to a significant percentage of users over which Google controls their email unless you switch to Google for email.
edit: A couple spellos and a clarification
Always from the same account
That doesn't necessarily justify the response, but IMO claiming "nothing was different" when something very clearly was isn't helpful.
It would seem you both agree that "[Google's] spam filters [are] either seriously broken or they tweak them to cause more people to switch to gmail."
Google marked as spam my 7th or 8th reply in a chain of emails with a prospective tenant (presumably) because my email contained instructions for sending me money. I assumed that they weren't interested anymore and had ghosted, but luckily we crossed paths at a grocery store or some such.
My emails continued to be marked as spam after they added me as a contact, individually fished each one from their spam folder, and continued to write replies. Weeks later (with no apparent cause) I started arriving in their inbox again.
That's a silly way to spin this. They can keep your inbox super clean by just not delivering any mail at all.
The way GMail is going, you'll soon only be able to get Google approved corporate spam using their email service. It took them longer, but they're pulling the same shit they did with XMPP. Open and standards compliant to begin with, and slowly converted to a bastardized Google only walled garden.
Maybe now's the time to fix it. If people don't get an email, suggest they use a better service.
But you won’t know if Gmail flags legitimate email as a false positive and you will remain happy not knowing Gmail is secretly losing your emails.
Not the case for me. I'm okay with some spam. It's not hard for me to delete few mails every day. But I'm extremely not okay with legitimate mail marked as spam. And gmail keep doing that. And that means that I must check spam daily. So basically gmail spam filtering does not exist for me, because I still must read at least all subjects. And that's dozens of mails every day.
From a few months ago. Self-hosted email getting dropped for seemingly no reason.
I still use Yahoo e-mail for chit-chat (I doubt that it's less secure). Recently it started spam-boxing e-mails from a person I've corresponded with for years (without trouble). Since I have them forwarded to T-Bird, it does mean I have to visit Yahoo more often.
Maybe that's the goal?
There are so many such 'narrow' cases, and they all just happen to work in Google's favor.
I'm getting about 2 spams per day, which doesn't really bother me.
Instead of just blocking bad actors why not work to prevent bad actors from being a problem in the first place?
Oh right, it would go against their business model.
The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.
We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.
I did a quick scan of rafa.eu.org and found the following:
- SPF is setup  with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.
- A DMARC record exists and has a valid syntax , but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.
- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).
- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh  to check this.
So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.
I wrote more about this here:
I would venture to guess that a substantial portion of people reporting issues delivering at small scale are sending from a low-reputation netblock. (Assuming auth is all set up optimally.) So it's not that it's not possible to send at low volume, it's just that most people don't know to look at the netblock, it's hard to evaluate netblock reputation, and it's not always possible to select a preferred netblock within a given provider (especially VPS providers with low-cost options).
The bottom line is: it's hard to run your own mail server, and much of the reason for the success of services like Sendgrid is a) it's easier to set up auth properly with most of them, and b) they actively manage the email reputation of their netblocks (though some do this better than others).
If you're approaching things from a practical perspective instead of making this your hill to die on, you just use an email service provider, and for the vast majority of senders, life is a little bit easier. There's plenty of ESPs to pick from, and there's a very good chance you can find a free one that delivers better than an average VPS or IP of a residential ISP.
I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time.
Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.
Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.
I’m waiting for the day they try to shut me down.
If you're wondering about the blacklist number, that's an error - the source site claims it knows nothing.
I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim -- I can understand that they often block legitimate mail like in your case, but that doesn't mean that everyone has the same issue. I have occasional issues sending to Gmail but most of the time they accept it and don't mark it spam.
Based on direct experience and >testing< (I have talked to about 20 people with personal mail servers about this, and got them to attempt to send me email, on an account that had never interacted with their domain before), 100% of their emails were spam boxed.
Lots of people test gmail with their personal accounts, and think that email to gmail in general is delivered.
In my experience, it basically never is if you are a small sender.
Where are all these magical people with these magical abilities? How curious that not a single one of them is able to post any kind of evidence to support these incredulous claims. And funny how my comments on HN are getting downvoted and hidden when I ask for proof.
Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.
In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam.
GMail will absolutely still spambox you.
Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.
An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc.
This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough.
A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does.
The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here.
> I did a quick scan of rafa.eu.org and found the following:
All the issues you list are non-issues, and merely the perpetuation of the security theatre.
SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either.
Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected.
Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.
1. Our ML models see that your emails spam? You're spam.
2. You're using Tor or a VPN? Here's 10 reCaptcha's.
3. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.
4. Youtube video includes education hacking tutorials? Demonetized.
Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
I get recapthca ALL the time for just not being logged on and deleting all filthy cookies on exit; and not "honoring" 3rd party cookies at all... and I don't care. Best part is that if you live in a country you don't speak the language you get google services in one of the country official languages -- so
what you should do and mark is often unclear. The next best part is a road trip through Europe (edit adding "?hl=en" to the url tends to fix the issue in most of google services)
>4. Youtube video includes education hacking tutorials? Demonetized.
History is a far greater offender.
>1. Our ML models see that your emails spam? You're spam.
Years ago, I recall my company hosted gmail started detecting CEO mails as spam... and he was pissed for not getting responses.
Seriously, you shouldn't have to. Your browser sends language preferences, doesn't it? Nope, IP trumps user preferences.
Had the same problem when using my job's VPN with an exit point in Germany (working from France). Very annoying.
Since http1.0... I have wanted to add another header -> "RespectAlHeaders:All-Custom-CrRafted", so the wimpy implementation of ip->country->language is duly removed.
Even if your language is English and your calendar is Gregorian, it will still display news dated in Hijri. No setting or option to change.
Now I've got a PhD on solving "traffic light" and "crosswalk" Captchas as a result.
I started to resent Google for this, except now I think twice before going to a web site that will give me these hurdles, so it ended up helping me to focus on my work. Now I appreciate the extra work.
Default Safari option for 3rd party cookies is: "from visited". Check if you have cookies set, if you do - they know you have been a good boy.
That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.
First time setting up an email server was quite a chore as I have no background in email, getting all of the moving parts to play nice required some patience.
My second email server used Mail-in-a-box scripts which I highly recommend. Zero to functioning email server in a few minutes. The slowest part is DNS propagation. Anyone comfortable setting up a domain name and the basic DNS settings that are invloved will have no problem self hosting email. The setup includes web-based client. I use k-9 mail client on my phone.
I find the catch-all email feature very helpful in creating ad-hoc addresses for spammy companies (eg car insurance). I can sign-up with firstname.lastname@example.org and it will direct to email@example.com.
I'm not completely de-googled, but I am consciously and gradually reducing my exposure to google, and other large networks.
At the very least it is worth a try.
edit:spelling, added url.
I do see mail going to spam though. It's never bugged me that much.
Please allow me to export notes to txt.
As for a more general download - what would you be looking for? A zip file containing all the notes?
A zip file with all notes in it would be fantastic. I just want to feel like my data is safe from Apple having a bug that wipes it. Read a lot of Catalina horror stories.
IMAP looks promising and might be all I need. Neither Spark or Apple Mail seem to let me export mail as txt but presumably there's a mail client out there that does in which case problem solved. Thank you for point that feature out, should throw it up on:
I use Fastmail scared that my emails will go to spam. But ideally I want to live in a world where my emails hosted on my email server will send like they did 10-20 years ago.
It's sad that it's no longer viable to have your own email server
Even though I have shut down my practice for the near future, I will still need to have information protected for several years to the Health Department's requirements for psychological records.
Email providers who lack this support will have difficulty gaining traction in the health world.
That's already the case.
In my opinion free email services either belong to other big corporations or provide lower quality service than gmail.
Few bucks a month add up and can be considered expensive by many.
I don’t like google and I pay for a different email account. I just wanted to remind that gmail may be hard to quit and that costs are subjective.
I pay around 3$/month for Fastmail, I don't think it's expensive, an email address is the primary communication method everyone uses nowadays and it's useful every day.
We could probably start to divert all gmail and outlook originating mail to the spam folders on our mail servers in retaliation.
I think that's exactly opposite. We need more competition, not forcing a private company to behave in some way that "we" (and who is "we"?) find acceptable.
It's time to break them up.
Google can exist, and people can choose to use Bing, DDG, Fastmail, etc., right?
The problem is that Google is not just Google the search engine, or Google the email provider, or Google the video service, or Google the website analytics engine, or Google the online office suite, or Google the ads syndicate.
Because Google is all of those things, it has accumulated too much market power, to the extent that they are market distorting. Bing and DDG (which is not an independent engine, but collates results) are irrelevant to a company's Internet presence. There is no such thing as SEO for Bing. Whereas Google can make-or-break you.
When Google the ad syndicate and Google the search engine collude (as it were) to place your competitor's ads ahead of your organic search result, a top result even, and to distinguish such ad results as ads only technically, by the narrowest of definitions, this compels you to advertise with Google the ad syndicate. Just as an example.
This behavior is harmful to the market (not to mention to users, but that's a different subject) and it is a given that Google the monopoly^Wsingle entity will continue to make such moves, whack-a-mole style so it's futile to squash any individual behavior even if you could. Ergo the reasonable solution is to break them up.
This ship sailed years ago, though.
Bitch about Google all you want, it's not like they decided this is how the internet should work. Prior to Google, my company integrated with reCaptcha because no other choice existed to reduce automated abuse.
I was surprised/disappointed to see this happen on HN.
Yeah well, they offer a free service and you use it. They own their network and they can decide what they allow.
It seems you're not getting the point. The problem is not for users of those "awesome" Google services the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account.
Google tags those as spam and refuses to deliver them, while always shifting the goal posts and providing sub par support (an number of essentially useless links) to the sender of the email to a Google account.
This happens, btw, also when the sender account never spammed and sent legitimate mails to the Gmail user for years.
So your snarky comment is completely out of line here.
It's not about people mooching free "awesome" Google services for free it's that Google is a rotten net citizen and is behaving like shit to the entire world.
It's funny to me that people in this thread are complaining about Google being large enough to serve as a de-facto authority, when the only way one could solve a "legitimate emails are being blocked" problem is to have a definition of 'legitimate' that would come from... an authority.
Google is dropping the ball but I don't know what the solution would look like that isn't just "Someone else be Google now." ;)
I don't use gmail these days (except as a honeypot out of laziness), but I get plenty of email from google hosted domains. The issue isn't (just) that people want to use these services and want Google to have more reasonable policies, there is also the issue of having to tolerate Google's policies because other people use their services.
Emails from Google servers are notable to me because there is no way to report spam (and the signal to noise ratio from Google is pretty low for my case). Google's opaque approach to email means that deliverability issues become even more tedious to deal with than they should be.
Maybe I’m weird, but I’ve never been in an AMP where I wouldn’t rather go to the actual side and use reader mode.
At start it has mark that search results based on results from yandex engine and crawled results
It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.
Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.
I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.
In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...
Aside: Microsoft bought Hotmail in 1997! How old is that account?!
There's a particular perfume company that sends spam to my gmail account every month. I've been marking it spam for at least five years, but somehow it always gets through.
Remarkably, Fastmail has gotten to the point where its spam filter works better than Google's for the kind of messages I get.
This site is very useful if you have your own mail server
At this point the only reason I'm still using Gmail is sheer inertia, and it's a question of when I get time to start untangling myself from it.
Meanwhile some Gmail addresses I have rarely have this problem, so it's not even consistent across accounts or domains.
The most annoying bit (and probably the first untangling from Gmail I'll be doing) is that I have my own domain hosted on Google Apps, but since my main Gmail mailbox is a different one, it's set to forward everything. It's meant to bypass all filtering and just forward. Or so the settings and support claims. It doesn't. I have to regularly log in for the sole purpose of marking things not spam.
Gmail filters don't seem to work anymore. I had filters and labels and it was great but I noticed they seems to be failing.
If you go down the rabbit hole of Google Gmail support you'll find filters don't apply to email ("conversations") already in your inbox. But they also don't seem to apply to new email.
Asking on various groups people seem to be giving instructions for options in GMail that no longer exist.
TBH, it's just easier to re-do the filters. There's also an option to make it apply to old conversations on a one-time basis as well.
Gmail even replied to me on Twitter saying the filters no longer work.
Screenshot of Gmail reply on twitter https://i.imgur.com/URhBeXn.png
I like using filters I don't see why such an important email tool is so awkward to use on Gmail.
SPF is not a "should" anymore, it is a must. Many major e-mail providers will outright reject your e-mail at SMTP level, unless you have correct SPF record. Rest will move SPF-less e-mail to spam on automatic basis. In this sense SPF does help you to get delivered, and avoid getting into spam bin, but your chances can only go from "100% spam" to "90% likely spam", not "0% likehood of being spam".
DKIM is not a major consideration for Gmail (or anyone, as far as I know) and have never been one. You will not magically avoid spam bin by using DKIM. Instead DKIM provides a great opportunity to botch your reputation with mail servers: as soon as you forward a e-mail (between providers or within the same provider) it will lose it's DKIM validity, and thus may get classified as spam (!!) on the basis of invalid DKIM signature. Boom! you are a spammer now.
Valid SSL cert and valid SPF are the only things, that have noticeable impact on Gmail's treatment of your email. DKIM and DMARK and largely useless and can only get you in extra trouble.
As for bulk email, it is going to be classified as spam until your domain and IP gain sufficient reputation. The only way to gain that reputation is to send slowly increasing amount of email for long time without being marked as spam. Being a rich business, Google's client, hanging on Google's forums, using Gmail as your mail server and having at least dozen contacts among Google's employees also helps.
I use SES from AWS and now my new user sign ups emails for email verification also end up in spam. It’s very frustrating because they changed some rule and now my low volume email sending is considered spam
..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).
I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.
My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).
Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.
This is why I like that my email provider (https://mailbox.org/en/) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.
so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.
depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)
if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.
the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.
I have no idea if it actually is, but it might be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.
That said, if you read far down enough in the thread, it's pretty clear that the OP has very, very unrealistic expectations about others' obligations to him, and very incorrect notions about how statistics work. The argument eventually turns to "well google has a lot of processing power at its disposal so I don't see why it can't use this in order to turn incomplete data into more complete data" and that's when I checked out :\
Edit: also this Brandon Long has the patience of a saint.
How things have changed.
Put another way, play stupid games, win stupid prizes.
But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.
It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:
"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - and that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."
They will accept an email, with all outward symptoms being normal, but it will not appear anywhere in the user's mailbox. Not even in the Spam folder.
This started few months ago when we were sending out our (rather infrequent) newsletter. 5th or so message to ...@yahoo.com address failed with
421 4.7.0 [TSS04] Messages from xx.xx.xx.xx temporarily
deferred due to user complaints - 184.108.40.206; see
(in reply to MAIL FROM command)
As I mentioned above, the "best" part is that none of these emails would actually make it into recipient's mailboxes. They disappear.
Better yet, all subsequent (non-bulk) emails have the same fate as do _replies_ to emails from Ymail users. This last bit is what makes this look like a shadow ban.
All attempts to contact Yahoo went unanswered, including those made through Gmail. Ditto for whitelisting requests through their web form.
If anyone has any ideas what the hell's going on, I'd be curious to hear them.
Slow down your send rate, make sure your server is configured correctly, and prune your lists.
PS. The list is immaculately maintained, the server is configured as properly as it gets and very conservatively rate-throttled. There's nothing obvious.
I honestly don't think they are nefarious, just overly insistent.
I remember email before Gmail's spam filters. Everyday was a task of deleting spam first then responding to emails.
But now that spam detection mostly works well for most people, it’s a forgotten issue. Now we have people on this thread questioning Google’s authority to filter spam because from their PoV, Google isn’t adding much value to begin with.
Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.
It's a very hard problem to solve.
With new technology, the laws of course lag. You see a wild west, then monopolies etc before it settles.