Hacker News new | past | comments | ask | show | jobs | submit login
Gmail marking email from me as spam (mail-archive.com)
475 points by cnst 43 days ago | hide | past | web | favorite | 393 comments



The email landscape is substantially more complex now than it was three decades ago. The poster seems to be frustrated that he is mostly getting technical explanations to his rather emotionally fueled question. He is literally phishing for an argument on the morals of a big company deciding he is a potential spammer without absolute proof. He made it clear that his opinion is that there should be a system in place at Google that he can petition for his reputation, and that emails that look like spam should say 'potentially spam' rather than send it to a spam folder. All this while sending emails from an origin that is very well known for spammers and whose actions depict a typical bot account warming up. (Poster claims to on occasion initiate personal conversations with people who have their emails publicly listed, such as an author from a news site he might disagree with). There is just no reliable system today that allows for someone to host their email servers in a spam haven, not use relays, act like a spammer, and expect not to get your emails sent to spam folders.

It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.


Rarely have I seen such concise and respectful way of describing the typical rants I see on HN way to often, bravo! I can only wish I had your writing skills.

> It's insulting to the folks that are pouring their time replying to what they think is a user in need of technical assistance, only to eventually stop replying when it becomes obvious that he only replies with questions of morality and treats most answers as if they were platitudes specifically attempting to dismiss his efforts to communicate.

Yes, I have been bitten by this many times. Every time I see a topic like this one I try to help that person by offering technical advice, basically doing my consulting job for free for everyone to read. And many times the post gets downvoted till its dead.

Slowly I learned that people don't want a solution to their problem, they just want to vent their (understandable) frustration about $BIGCORP. But it still hurts to get downvoted when you just try to help.


This is essentially HN with every single Google related post nowadays. Good or bad, people hate Google, they hate it the way they used to hate Microsoft, they see everything it does as being nefarious.

People ignore the email spammers, SEO spammers, automated password crackers and blame google for the solutions they put in place to combat those things.

It's impossible to reason with them, honestly I think it's just best to ignore posts like this.


I am not an email spammer, an SEO spammer, an automated password cracker, or a jaded failed Google canidate.

I have my mailserver configured in pretty much an optimal way, with all the special bells and whistles, have never sent spam, and never will.

Gmail refuses to deliver my mail, so please don't lump me in with malicious people and then say my concerns should just be ignored.


That may be true, but unless you post relevant technical details, that means nothing to strangers in the Internet.


Have you considered using a full TLD and hosting and/or relaying through a reputable service. I've used mailgun and sendgrid both for moderate accounts for outbound email and didn't have any issues. There's also hosting options for mail ranging from fastmail, to google, MS and others.

In the end, TFA is from a subdomain that seems to have a lot of spam, and to avoid it may mean using A commercial option, if not a specific commercial option. It sucks.. but the mail relays I mention above are relatively cheap and easy to properly setup with limited issues on the other side.


I am not interested in a third party handling mail for me.


If you were doing everything right it would work, unless you sent email from an IP known to send spam in the past. That doesn't necessarily mean you did, but it does mean your IP did at some point. Given the saturation of the ipv4 space this isn't exceptionally unlikely. In any case, your post is contrarian and doesn't offer any substance aside from the fact that you are angry. That's kind of saddening because that was already pointed out as exactly the problem.


[flagged]


Believe it or not, its possible to carefully look into things and check them, but I understand its much easier to be smug and assume that its my fault.


Judging by your username I don't expect you to share the domain name here, but if you need a second opinion, send me a PM and I'll analyse the domain for you.

There is usually some misconfiguration somewhere that goes overlooked. Happens to the best of us :-)


I've thought about setting up a mail server on a DigitalOcean or Vultr VPS without any experience with mail servers (for person use). Whenever I read opinions about operating a private mail server, there's usually a few people that express great satisfaction in doing so, and say that it's not too difficult. However, there are also many more people that say it's not worth the effort since there's maintenance to consider, complex setup, and the cost of email providers is usually the same or less than running your own server.

What is your opinion (or others' if they'd like to chime in)? If you enjoy running your own, are there any guides in particular that you'd recommend? Does the future of email look even more prohibitively complex for self-hosting?


Self hosting has become harder due to email abuse (spam, phishing, impersonation, etc) and the measures against this that have been introduced.

Following a guide to setup a basic email service is quite easy to do, but after a while the lack of proper implementation and maintenance of the standards will start to cause deliverability issues. Those are frustrating and most end up writing a long rant on why Google is evil because their email ended up in spam. When in fact, it usually is a misconfiguration, but those can be really hard to detect.

IMO there is still great satisfaction in running any internet service yourself. But, if you rely on this service to work, it just no longer makes financial sense to do so.

It takes resources to keep any internet service running correctly. Even the simple stuff can quickly get overlooked, I have seen so many self-hosted email servers that have expired certificates (like with the domain of the OP) or lack support for any of the modern security extensions such as DMARC or MTA-STS.

For most businesses my recommendation is not to run it yourself. Like you wouldn't run your own DNS service anymore. Of course it can be done, and in some cases it makes sense to run stuff yourself (usually when operating at a very large scale). But it almost always makes more financial sense to use a hosted email solution.

We haven't written a guide (yet) on setting up an email server with all the bells and whistles. But if you want to read more on all the email security standards, have a look here: https://www.mailhardener.com/kb/


Thank you so much for the detailed reply. I'm fairly new to programming, development, Linux, the command line, etc etc, and so am always on the lookout to learn something new. My day job is web development, but I'm looking for something that I really want to dive deep into. Email seems to be a whole universe of its own! From what you've said, maybe I'll try it once to learn a bit more someday, but for now I'll let sleeping dogs lie and keep G Suite as my custom domain email host. Thanks again!


Honestly, running an E-Mail is so tedious that I just allow Migadu to manage it for me. You have to deal with configuration (as the original poster said) and run into issues with systems just blocking you outright (especially Outlook). There are some utilities or Docker images that make it easier, but in the end its not worth the hassle. I still think domain-based email is way better than allowing G-Mail or Outlook to host your email though.


When you say domain-based, you mean using a email-hosting provider with a custom domain, is that right? I currently have my custom domain set up with G Suite for email and am thinking that's probably good enough.


I ran my own for a number of years... it was a pain in the ass and took too much time for what it is/was worth. I'm a frugal man, but it's just not worth it to me. I now have several relays pointing to my gmail and/or a custom outlook.com domain (grandfathered free account). I've thought about putting something up, but it's a headache and a half.

For delivery, if you are really just using it for personal use a mailgun or sendgrid account can allow you to handle most delivery issues, while still handling inbound.

Which still leaves the other side, actually dealing with inbound mail issues which is just painful with either too much or too little filtering. The likes of google and MS have teams of brilliant people to handle dealing with spam, and despite some little guys caught in the middle, is still far better than I could ever do on my own.


Thanks for your take on it. I'm fairly new to the world of programming, development, Linux, etc., so I'm always looking to learn things and make something of my own. A mail server sounds very neat in that regard, like building your first computer, but it sounds like in reality it's best left to the experts. I currently have a G Suite account for my custom domain, so that's probably good enough.


I don't mean to discourage anyone who wants to experiment... It's actually really interesting with a lot of options in terms of running your own. It helps to learn things from the security aspects, to the use of tools for spam, greylisting, dns lists etc. On the flip side, I wouldn't do it for my primary email again.

In the end, if you want a career in IT, or find it interesting, I'd say go for it. I've often thought about building my own end to end open-source mail service aimed at ease of administration. If I was ever rich enough to not have to work, that's one of the things I'd probably do after a few months off.

I'd suggest starting with a secondary domain and using Mail-In-A-Box on a VPS or smaller Cloud host. It's a decent starting point, and there are many other options. One of my favorites is SmarterMail (commercial, windows only though). I also ran a BBS for a number of years using Synchronet, which does email/pop/smtop and even nntp for group messages. I did have it configured with SendGrid at the time for outbound for a while, which handled delivery issues for me.

You will have headaches if you take it seriously and/or use it as a primary service without using a delivery service like MailGun or SendGrid, and even then you probably will have other headaches. My point isn't really do discourage so much as let people have a more realistic understanding. The issues the OP has are real. However, there is so much junk from bad actors that the well is poison.

cnst 43 days ago [flagged]

If this is the kind of technical advice you provide, which merely assumes that Google can do no wrong, and the user on the other end is incapable of pressing a button on a yet another website that checks email configuration, merely to confirm that the configuration of their mail server is not in any way incorrect, then it's not a surprise that you get downvoted into oblivion, according to your own statement earlier in this thread.


"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."

https://news.ycombinator.com/newsguidelines.html


I can't provide any technical advice unless the op shows their config. I am downvoted because this community is already biased thinking Google must be at fault and the configuration can't be wrong at all ?.


You were surely downvoted for posting unsubstantive and bilious comments, which don't belong here.

Please make your substantive points more thoughtfully.

https://news.ycombinator.com/newsguidelines.html


I would say the community is biased the other way around, where everyone always thinks that it's the little guy that has invalid configuration, instead of Google being at fault.

What sort of config do you require from the OP? Their domain name is public and uncensored right in the top of the post. Their configuration has been cross-checked by many people so far. A lot of folks claimed that it has issues (like a `?all` in SPF, or `p=none` in DMARC, or some other nonsense that any spammer would easily get right), but those issues have been rebuked as not being significant to the issue at stake, plus, there's absolutely no confirmation that fixing these "issues" would resolve the problem, either.


And is your server sitting on the same IP range as those that do? Do you know?

Google provides a service to its users, they don't provide a service to you.


Isn't receiving legitimate email and delivering it to Inboxes part of the service it provides to its users?


Filtering SPAM mail with an acceptable false positive and false negative rate is part of the service.

Unfortunately for you, the e-mail providers' customers (the users receiving the mail) get to decide what is an unacceptably high false positive/negative rate --- and not you. If e-mail recipients get too much SPAM or if they get mail that they want landing in their Spam folder too often, then they will switch. This provides the economic incentive for mail providers to Get Things Right.

Unfortunately, there is so much spam out there that if you are too relaxed with letting marginal e-mails (or e-mails from marginal network neighborhoods through) the false negative rate could result in so much spam in users' inbox that they will get mad, and switch.

However, it can happen that the email provider can be too sensitive, and there can be too high of a false positive rate. For example MIT recently tried relying Microsoft's spam filtering system. Way too much stuff landed in the spam filter, and while you could go to the spam filter and manually mark e-mails as "not spam" and eventually Microsoft's ML algorithms would hopefully figure it out, the false positive rate was too high and the MIT community (especially the ones with the real power, e.g., the professors) rose up with an outcry, and MIT abandoned using Microsoft's Spam filtering service.


My experience is that the email provider's customers have very little say in the matter. I deal with complaints from Customers who use Office 365 and Gmail about false positives regularly. They aren't MIT-profile Customers, so their complaints are ignored. The complaint usually turns into "Why can't this sender just use Google or Microsoft for their email like 'everybody' else?"


It depends on the customer. If you're an Office 365 customer who is a major enterprise with huge numbers of paid users, and your CIO goes golfing with Microsoft board members, you have a huge amount of power. Similarly, if you are an MIT Professor you have vast amounts of power.

They can be pile driving right next to the machine room, and causing disk drive errors, and Physical Plant won't care if the I/T folks complain. But if some Sloan School professors complained that it was disturbing their cogitations, it immediately stopped and Phys. Plant called a "stakeholder meeting" and it was really obvious who had the power.

All customers are equal, but some customers are more equal than others.


Why would you assume that?


"People irrationally hate X" may be perfectly true, but is not a reason why any particular, or most, or the most significant, criticisms of X are wrong or should be ignored.

Even the particular people who irrationally hate something are obviously likely to also run across the good reasons for hating it.

It's not reasonable to say "criticism X doesn't count anymore because we've heard it too many times".


That can be annoying...

Pro tip: start by briefly acknowledging the frustration, then offer the technical solution/workaround.


Even technical people are people. This tip can help defuse the frustration, making the recipient more willing to listen to the technical solution.

People forget sometimes that the person on the other end of the screen isn’t a computer that is completely unmoved by silly things like emotion and frustration.


I don't think OP stop replying, he seems pretty active on another thread of the same mailing list concerning another big player (MSFT) filtering out some messages for some reasons. https://www.mail-archive.com/mailop@mailop.org/msg09066.html

As a hobbit* , I'm actually pretty happy this whole conversation is starting to gain more traction.

My domain is always green on every test I could run (mailtester.com & all) yet some of my mails go to the junk without any warning. At least warn postmaster when you decide to block a domain because of bad neighbour/hosting!

* This specific reply of the chain is a must read and is where learned that I'm actually a mail hobbit. https://www.mail-archive.com/mailop@mailop.org/msg08958.html


> I'm actually pretty happy this whole conversation is starting to gain more traction.

This exactly. I've been a Gmail user since the day when they've required an invitation in order to signup, and I'm not quite a Google hater, either, but it's extremely disappointing to see the number of people who simply defend Google against the wrongdoing irrespective of whether or not the complainant has a valid complaint, like in this very case.

What happened to the whole notion of believing the victims? Why no matter how hard you try to explain your setup, and the things you've tried, and that all the automated tests pass, that you still get blamed to be the culprit for the issues you experience when dealing with someone who has a monopoly on the service at stake, instead of considering the possibility that maybe it's Google Mail having the monopoly on email that's an issue here.


Oh, I do think that Google is absolutely more aggressive than they probably should be on filtering... On the flip side, there's very little other choice. The VAST majority of email comes from a handful of hosts/providers and everything else has a much higher noise to signal ratio and gets much more scrutinized period.

Also, if you aren't sending a lot of mail, consider relaying outbound through mailgun or a similar service (I used to use sendgrid myself). Nearly zero issues when doing this on the lowest level paid account. That said, inbound spam was so painful to deal with I just gave it all up.


Check your neighborhood :) you can use Talos to see the reputation of your IP block and ASN:

https://talosintelligence.com/reputation_center/lookup


Even after reading that, I'm not sure what an email hobbit is. Someone who just wants to live in a cozy hole in the ground with a few nice email amenities?


It was an extended metaphor categorizing an ISP's (or a cloud provider) customers into multiple buckets: Kobolds, Lizardmen, Hobbits, and Princesses.

Princeses were the customres which sometimes required a lot of customer support, but paid really well because they purchased a lot of services/VM's. Hobbits were customers who basically minded their own business, and didn't guy a lot of stuff, but didn't cause any problems.

Lizardmen were customers which had possibly clueless marketing departments that would occasionally send e-mail marketing campaigns which crossed the line, but which also sent plenty of legitimate e-mails, and which were mostly trainable after the ISP/cloud provider smacked them on the nose with a rolled-up newspaper --- and Kobolds were spamvertising companies that did nothing but send spam, and were constantly switching ip addresses / VM's, and knew d*mn well that they were doing something evil, and were not trainable.

The problem is that if the ISP / cloud provider has the vast majority of their customers being Kobolds and Lizardmen, it becomes economically hard to just cut off the bad actors, because the vast majority of their customer base are bad actors. If the provider originally had a good mix of all of these customer mixes, but wasn't proactively cutting off the Kobolds and educating the Lizardmen, then their network block would start getting a bad reputation, and all of the major e-mail providers (not just GMail) would start blacklisting the netblock, or at the very least, treating any e-mail sent from that network block as a strong SPAM signal, and this would cause the other customer types to find other providers.

So if you happen to be a customer of such a network/cloud provider, you could try complaining to the provider, but they are really in between a rock and a hard place, because if you're the 1 good hobbit and all of the otheer customers for that provider are Kobolds and Lizardmen, what is the provider supposed to do? You're probably better off switching to a better provider. Unfortunately, the better provider may also cost more, but you get what you pay for.


My impression from that was (almost) everyone is a hobbit. Hobbits being normal people who don't want to send marketing spam.


But still want own their own domain so that they are not tied to any internet king.


Basically. His fundamental issue is that Internet services that lack an authority framework organically bred "ghetto" infrastructures, where things work worse because a person lacks access to "the right kind of people" in a system that's supposed to be democratized. He's not wrong, but it's not like someone sat down and decreed some email sources should be untrusted for fun; the system started out very implicitly trusting, we added a pile of humans to the system, and oops! They acted like humans, took advantage of other people, implicit trust broke down, and now there are "good" and "bad" online neighborhoods.


I'm curious how you define "democracy". Email delivery is very democratic, with each person having the power to choose with whom they associate and who they choose to delegate decisions to. Anyone who doesn't like Google's spam blocker can advertise an alternate address on a mail server that welcomes all submissions (or host a blog, or join a freely open IRC channel,...)


It doesn't really matter if he or the gmail user initiates the conversation, in my experience. Probably the only reliable way to communicate with gmail users is to send them 5xx response right away when they first contact you, and inform them there about the situation, and that they should check spam and why, tell them to resend, and let the other messages from the same gmail address through (maybe repeat this from time to time so they will not forget to use their e-mail properly and check spam folder). Let them have some of the fucking time waste their provider causes others.

Because it's really infuriating if someone writets to you, you put time to help them with something, and their demented (did I say it's infuriating? :D - also what kind of company can't detect something is a response, if you even quote the original email, it's ridiculous) e-mail provider trashes your response.


This is actually a great idea. We should implement this two-step approach for anyone trying to email us from Gmail.

Sadly, I'd imagine some users don't really have a choice, e.g., an employee whose company is using Gmail from G Suite for its email, so, it'd be an uphill battle for sure.


Maybe, one other thing where this would have to have a shortcircuited is when I initiate the contact, so that people responding would not get hit by this, even if their reply is the first e-mail from them sent to my server. So it should be fairly smart.


You could try that. But I'm not sure so many people would think that corresponding with you is worth the trouble.


I can't really care less about those people. Most people I'm talking about are people who're contacting me because I already provided them quite a bit of free service via my free software.

So if someone can't be bothered to check so that I'm not wasting time responding to their trash folder, that's a good filter in the first place.

Anyway, better thing might be responding by adding a (F)AQ item publicly, and just sending a link.


> He made it clear that his opinion is that there should be a system in place at Google that he can petition for his reputation

The common thread among many of the bad-Google stories I've seen in the last 10 years is that Google has made a deliberate decision to not give two shits if they accidentally hurt a small minority of their users. People get messages out of the blue that essentially say "We've just wiped out your email account, we're not telling you why and there's no one you can appeal to." I don't know if this particular guy is on the level or not, but it's not credible that all of these people are secretly spammers and scammers. Occasionally you can get things fixed if you know someone at Google or your complaint goes viral, but that's your only option.

I'm sure someone at Google did some math and concluded that the ill will of a few customers costs less than maintaining a support department. That's not an excuse. I don't think we should expect Google to never make mistakes, but we should absolutely expect some avenue to correct those mistakes.


No idea how well it is or isn't correlated, but a note on twitter tagging the appropriate dept at google usually sees the issue fixed within two days, though no actual feedback and likely coincidence.


Brandon Long, is as ever, a helpful and respectful voice of reason on the sometimes fractious Mailop list


Sometimes fractious? Ha. There's so many cranks there that 95% of the messages are not worth reading, even if most are well-intentioned.


I’m going through the thread and the answer comes down to reputation...is that really a technical measure of the issue rather than a social one?


I would argue in this narrow case - I would not point to nefarious undercurrents.

Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.

If you can get into people's email inbox, there's still quite high conversion rate - better than clicks/videos on a pure percentage basis. Even the author wants to reach out to people in a cold fashion.

As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.

As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on.

I can't have it both ways. I would not be on Gmail if it was like yahoo. My goodness... my parent's inbox, on yahoo, makes me anxious.


> Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.

Google's spam models freaking suck donkey dick. I use Fastmail but my university uses Google for email. A service we use (Piazza, class Q&A) sends notifications via email. Over the years I have gotten probably about 1k email notifications from them, and replied to a few of them too. Google has recently started marking about 5% of their notifications as spam. 95% of the notifications come through; 5% get rejected. There's no discernible logic to this.

There are organizations and people within the university which have me on their mailing lists; with them too 90% of the emails get through but 5% get randomly marked as spam. And they're perfectly normal emails that Google just decides to move to spam. Even with people that I have had email conversations with, it just spams random stuff.

Every couple of months I check my Fastmail spam folder just for the lolz of seeing what's in there. With Google I have to check spam every day or I'll miss important stuff.

There are tons of people who have posted very similar experiences on HN. On this thread I just saw another person comment that Google's own emails go to spam - I've observed this too on my personal Gmail account which I occasionally check.

The spam filters have a tough job, but other companies are succeeding at keeping all the spam out without a gigantic false positive rate. Google's filter is hilariously bad.


I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.

I've seen this with my SaaS. We ONLY send transactional emails like purchase receipts and login links (using postmark). We're very intentional about not sending anything remotely resembling marketing or spam, and sending as little as possible. Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.

I've noticed that in the past year or so gmail has made hitting the spam button more explicit and now have a confirmation message. Obviously they did this because some people were simply mashing that spam button instead of the delete or archive button.

I don't even blame the users. Most don't know the impact it has. But bad actors sending tons of spam have trained them to be cynical of any generic form email.


> I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.

Google does put some senders often into spam for me, but not always. I'm absolutely sure I've never reported them as spam, on the contrary: I've marked their emails as "Not Spam" when I caught them in the spam folder.

That had, so far, no influence on the filter. Those aren't "weird" emails either, they are in my language and individual communication, not mass mails.


>Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it.

Maybe gmail shouldn't have the spam button right next to delete :)

At least that's better than web Outlook's invisible, until you click on the sender, microscopic disposition buttons. I can't count the number of times I've deleted an email instead of opening it.


It might be because the service sends both spam emails and legitimate emails. Many (maybe even most) sites do this. If this person is getting a thousand emails over the course of a few years, how on Earth are all of those legitimate, especially when they've only replied to a handful of them, implying the majority of emails can oe safely ignored.


Yet I still get the occasional user who will flag their purchase receipt as spam when all they really mean to do is delete it. They don't get the difference and don't realize that it matters which button they click.

I have a relatively high profile email address (first initial and last name) and I almost daily get at least one form of "receipt" from a transaction someone made where they entered my email by mistake.

Sometimes I'm in a bad mood and I'll mark anything I see like that as spam partly because its frustrating and partly because I suspect a good number of these emails actually are spam masquerading as receipt/login/etc emails.

And I just don't have the energy to figure out which are legit or not.


I wonder how much bad spam marking happens maliciously. Sign up lots of accounts to your competitor's mailing lists, mark everything they send as spam so that hopefully their email will not make it to their customers giving you more of a chance.


> I'm not defending gmail, but I will say that some users simply suck (unintentionally). I bet some of your peers are just hitting the spam button instead of simply deleting the email, generating a bad reputation for the sender.

And yet other companies with spam reporting features are able to maintain a lower false positive rate with their spam filters.


I'm not sure that other companies near the scale of Gmail have a lower false positive rate. Sure something niche like fastmail which is going to have, on average, much more tech-savy users, will have more accurate user-reported spam filters. Yahoo / Microsoft don't seem to do any better than Google here though, in my experience they're both actually worse.


I don't think it's always the user. The gmail and outlook UI makes report as spam more prominent then delete. In addition it's easy for accidental inclusion in a users batch spam report because of poor multi select UX.


What would be nice is a useful mail ML filter which would scan mail flagged spam to trigger a "are you sure this is spam, do you want to delete instead" question.


May be you should not send purchase receipt by default? I don't know why everyone does that as I don't care about it at all. If it's not required by law, add button to your UI which will mail receipt (or better just let user download it) and that's about it. While I'm not marking those receipts as spam, I'm sure that some users would do that as it's unwanted mail for them.


We actually have users contact our help desk asking for the receipt not realizing it's already in their inbox or spam folder.

I've considered adding an opt-out checkbox during checkout but I honestly doubt it would ever be unchecked and if it was opt-in we'd have users contacting support asking for it because they didn't realize they had to check the box. Either way it's one more toggle on the UI which I'm not a fan of unless absolutely necessary.

I'm curious if you've run a business and experimented with not sending receipts, and if so how it played out?


Please keep sending email receipts by default.

The purchase history on website disappears, logins get forgotten and so on. Only receipts in the mailbox stay forever, as I want them to.


I would find it odd not to receive a receipt. A receipt email should not be classified as spam. Anything else I would say would be spam.


You should be able to set a filter to catch these before they get spammed out of things you want to receive aren't getting your inbox.

Also, a lot of users do perceive receipts as spam.


Yeah, gmail is really not great. I host my own email but use gmail for work. I hate everything in gmail, from the UI to the lack of really simple way to filter thing (like sieve) and to the totally random behaviour like you described. In the end, I don't really read emails on gmail anymore because it's just a huge pain.

My own server has a good reputation, but it meant setting up SPF, DKIM, DMARC etc.... Where it was the most painful was with microsoft email (I don't know the name of their service now, still live maybe). By default mails goes to spam, but really quickly they goes in limbo if people don't respond (on the smtp side the email is accepted but the recipient never see it, not even in spam). On new microsoft accounts I was always able to send a few emails in spam before it went into limbo. I ended up creating ~20 different accounts on their system (it was before they enforced giving a phone number) and had random discussions with myself for a while which was enough to build a reputation and not being put in spam anymore. I don't know if this behaviour was intended or not, I try to get in touch with someone from microsoft but couldn't find a way. Since then I do strongly advise not to use their system because that makes it really unreliable.


I have exactly the same problem on my personal email server, ms silently drops my email! Their server replies with OK, but the email is nowhere to be found in the recipient’s inbox. Even if the conversation was initiated by the microsoft account and I merely reply to it, the email still disappears.

I’ve got everything setup properly, all test websites come up green, and google receives my email just fine. SPF, DMARK, DKIM, PTR, HELO are all setup correctly. This isn’t an isolated problem either, the ms forums are filled with people reporting similar problems.


One thing I've noticed too, make sure that your time is synchronized with an NTP server. Even with a good reputation , MS drops emails with no error if you're a bit off.


"I have exactly the same problem on my personal email server, ms silently drops my email! Their server replies with OK, but the email is nowhere to be found in the recipient’s inbox."

Slightly tangential ... I heard recently that "proper" in-line responses (ie, not top-posting) is somehow penalized by Microsoft (or Exchange online or Outlook online or ... ?) and was advised to stop my very proper 1990s style in-line email quoting/responding and just top-post to avoid either non-delivery or something like the body of the email not being shown ...

Or something ?

Does anyone know what I am referring to and can shed some light here ?


Yeah, I only bottom-reply when I know (or can reasonably expect, e.g. on a mailing list) that the recipient is using a mail client that's not entirely brain damaged. Otherwise, I'll either top-reply or (if I really do want to respond to things in-line) actually say "See below for responses in-line" to clue in the recipient that yes, there's actually content in this email.

But yeah, that's usually more an issue with the mail client (or perhaps the mail server if it's being exceptionally stupid and modifying emails before final delivery). Top-replies became the norm for some god-forsaken reason, and now clients treat the quote and everything below it as entirely optional to display.


Same expirience here, but I must say it is becoming better since I have reverse resolve. Previously every mail went into spam. Regardless of SPF and DKIM.


I've run my own personal email for around 20 years at this point using the same IP address, and have set up all the bells and whistles (SPF, DKIM) over the years as they became in vogue. To my knowledge, I haven't had any issues with getting caught up in Google's or MS's spam filters, but I have to wonder if that's simply because I don't send a lot of messages and for the most part reply to emails rather than being the initiator of a conversation.


MS is really bad at this, a few web forums I am involved with are unable to sign people up at all.


Absolutely agree with you. Many people on HN suffers from Stockholm syndrome when it comes to Google. If Google wanted to be a customer focused company instead if being an ad revenue focused one they could expose the knobs to you to fine tune the spam filter. They do not care about users, Gmail does not even have a customer support line that you can call when you ended up in the wrong side of their automation code. I am in the middle of moving out every email address from Google asap. I just realized how disgusting their way of operating is and I do not want to continue contribute to this problem by using their service.


Just for fun I checked my spam folder as I read this.

I found an important email (from 3 days ago) from a recruiter asking for my CV in it.

Yeah, Google's spam has erred too far towards making false positive errors.


It's curious that people have such diverging experiences with the spam filter. I think the claim that Google's filter is objectively worse should at least be supported by some evidence if you're going to make a bold statement like that.

As another data point, for me and friends I it beats competitors by a large margin.


As a normal gmail user you won't notice this, all you see is that you get mail from your friends that you always mailed with. If you register on popular sites, their mail will also work. If however you are sitting on a new domain, sending mail to an old friend on gmail, it will very likely end up in the spam folder unless you are using a big provider that has reputation with google already.

I guess it's another case of survival bias. If it works you won't notice that it does not work in all cases.


Even a lot of “popular” site stuff ends up in spam. I basically have to regularly check my spam box, which previously I would look at a few times a year when a password reset or something didn’t show up.

Anti-trust regulators should specifically look at situations where Google began directly competing with other companies and those other companies gmail messages are being put in spam while Google’s messages are not. Whether or not it is intentional, it is a problem if Google is whitelisting their own stuff but not competitors.

Ultimately what is going on here is Google is using more deep learning for spam detection. Your friend’s email ending up in spam may be a false positive. It may also be a test, to see if you notice it.


If you never look in your spam folder you’ll never notice how utterly bad it is. Which is apparently the case for most people.


A lot of people don't even know there is a spam folder


So I own a very short, easy to type domain, and maintain a wildcard. By volume, I probably get 60% spam to 40% content. I apologize to all the hard working vendors being fed into my spam list, but there's a secret: double opt in.

As a bonus, if you require opt in, I won't be able to cancel your car reservations with budget ROBERT.


Your comment prompted me to go through my GMail spam folder. I noticed 4 emails (out of 100) that weren't spam, including one from Blue Origin (!), 2 from a mailing list that's usually delivered to my Inbox, and 1 from an online shop (confirming my purchase).


Did the same thing on work email and found only messages from gitlab. But only 10% of those messages were marked as spam the others were in inbox so I did not notice. This convinced me not to trust gmail antispam.


I didn't even know there was a spam folder hiding in the collapse ui.


Piazza absolutely sends spam about the recruiting side of their platform, whether you like it or not.

I absolutely click spam on every single one of those pieces of mail, and I'm glad that they are being penalized for it.


I've noticed this too. I've stopped telling people that their emails are getting spammed because I've determined it's not their fault, that google just randomly spams emails coming from lists. It's the main reason I'm working on moving away from google back to my own mail server.


"As a business owner, I'm continuously having to do things so I don't end up in spam. Less images, specific wording and the list goes on."

We[1] run our own mail and DNS servers and, as you might imagine, have never sent a single HTML email or any email with any kind of "element" in it, etc.

Just plain, ASCII text.

We've got a 15 year clean IP, perfect score on the "mail config checkers" and almost every single email we send is in response to an email from that sender.

And yet ... often enough to make me anxious, we get marked as spam.

[1] rsync.net


We have the same issue. Online surveys being sent as invites to 100s of 1000s of people per day and although we are good at spotting SPAMMERs setting up free accounts, obviously people will always receive an invite they might consider SPAM and we get derated. Persnally, we find Yahoo is the worst for blocking.

It would be good to have a central registry of mail forwarders so that SPAM is not detected against registered IPs and then issues can be followed up directly with anyone who abuses the system. Do you really want to greylist Amazon SES because one of its users sends out spam?



Unsolicited online surveys are always spam. I will always mark as spam with impunity.


Sending text-only is probably a weak spam signal as far as Gmail is concerned; for the majority of mail to the majority of recipients (though yours may be exceptions), HTML email - even if very basic - offers better layout/design options and better readability.

On top of that, of very old-school users on Mutt or Pine who prefer text over HTML, I'd think a significant portion of them don't have their mail pass through Gmail's filters.


But it's not about how good the Google's ML models are, you're missing the point! It's about Google being the huge many-headed monster with tentacles.

You can't use part of Google service without being entangled in their new protocols, the way they code and build things, run operating systems or servers, browse the web, and so on. You have to run your business the Google way now. The most scary part of this - the monster is completely deaf. Cry as much as you like, it's not going to listen. Even if you bring thousands of gifts to it every day, you can be erased from the web in an instant, and it won't even notice.

It's not too evil (yet!) but it has so much power it's scary.


Not so much "deaf" as "huge." They're supporting the needs of a billion users with sub-50,000 engineers. Anything that effects only 1 user can't pull priority from the needs of the many. Anything that effects only 10, 100, 1000 users, etc.


> As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.

Interesting how different things are for different users. Gmails spam protection isn't working at all for me. Confirmation emails from reputable sources get into spam, while really obvious spam messages get into my inbox. I mean.. really, really obvious messages. With "dfdf" a thousand times at the end just to make the message longer. Or these porn-bitcoin-scam messages.

It's ridiculous.


I spoke to somebody the other day about highly targeted ads that come from snooping into too much private information. The lady was happy about it because it helps her find things that she likes faster.

It's just interesting how perspectives differ on this type of thing.


I wish spam filters would automatically move all marketing/newsletters to spam


They sort of do, if you opt into Gmail's tabs feature.


There is no way that Google would suddenly start to mark as spam continuations of threads that were perfectly fine between senders that have been in contact for years unless their spam filters were either seriously broken or they tweak them to cause more people to switch to gmail.


I'm afraid you're wrong here, Jacques.

When you refer to specific threads going back and forth this may be. But here's my counter example.

I sent emails to my girlfriend's gmail address from my own domain hosted on a mail server run by a reputable company for years with no problem, whatsoever.

Then, on vacation in Thailand, my rather elaborate trip reports for which there's really no plausible explanation that they could be spam, suddenly bounced.

Sent from the same domain from which I sent mail for years and Google suddenly branded me a spammer and refused to deliver those mails even to her spam folder.

The auto-reply from Google is about as useless as you would expect. There are a few links, which are neither informative nor do they help to resolve the issue.

I worked around the problem by sending mails from my own Gmail account, but it left a really rotten feeling around the lines:

NICE COMMUNICATION YOU HAVE GOING THERE, WOULD BE A SHAME IF SOMETHING HAPPENS TO IT!

So I guess your assertion is wrong. Our exchange went on for a minimum of four years. Always from the same account and suddenly I was deemed an evil spammer.

It really sucks.

or they tweak them to cause more people to switch to gmail.

That, actually, is my suspicion. It's easy to brand that as "protecting their user base", but in reality make it so cumbersome that you're not able to send mail to a significant percentage of users over which Google controls their email unless you switch to Google for email.

edit: A couple spellos and a clarification


    Always from the same account
...except for the part where they suddenly started having a source IP in the headers in Thailand.

That doesn't necessarily justify the response, but IMO claiming "nothing was different" when something very clearly was isn't helpful.


Exactly. Look at it this way: an account with good reputation for years, suddenly showing up from Thailand instead, is exactly what a spammer hijacking the account would look like.


To clarify it, the GP is claiming that Google is acting in bad faith, not that you are doing anything wrong.


> I'm afraid you're wrong here, Jacques.

It would seem you both agree that "[Google's] spam filters [are] either seriously broken or they tweak them to cause more people to switch to gmail."


You can fix your spelling errors but you still failed to properly read what you were replying to.


Regarding a specific thread going back and forth (span of a month or so, not years):

Google marked as spam my 7th or 8th reply in a chain of emails with a prospective tenant (presumably) because my email contained instructions for sending me money. I assumed that they weren't interested anymore and had ghosted, but luckily we crossed paths at a grocery store or some such.

My emails continued to be marked as spam after they added me as a contact, individually fished each one from their spam folder, and continued to write replies. Weeks later (with no apparent cause) I started arriving in their inbox again.


> As a gmail user, I 100% appreciate keeping my inbox extremely clean.

That's a silly way to spin this. They can keep your inbox super clean by just not delivering any mail at all.

The way GMail is going, you'll soon only be able to get Google approved corporate spam using their email service. It took them longer, but they're pulling the same shit they did with XMPP. Open and standards compliant to begin with, and slowly converted to a bastardized Google only walled garden.

Maybe now's the time to fix it. If people don't get an email, suggest they use a better service.


Embrace, extend, extinguish? Gee, where did i hear that before?


> As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.

But you won’t know if Gmail flags legitimate email as a false positive and you will remain happy not knowing Gmail is secretly losing your emails.


> As a gmail user, I 100% appreciate keeping my inbox extremely clean. Google's spam filter is a welcome addition.

Not the case for me. I'm okay with some spam. It's not hard for me to delete few mails every day. But I'm extremely not okay with legitimate mail marked as spam. And gmail keep doing that. And that means that I must check spam daily. So basically gmail spam filtering does not exist for me, because I still must read at least all subjects. And that's dozens of mails every day.


This is exactly the viewpoint I have as well. I have missed some very important emails before, such as instructions for an interview, acceptance emails, etc... They don't even show up in Gmail's search system, because it doesn't search through spam. This has forced me to look through all of the spam I get daily, examining each one to make sure I'm not missing something important. Gmail may as well not filter spam at this point.


https://www.tablix.org/~avian/blog/archives/2019/04/google_i...

https://news.ycombinator.com/item?id=19756125

From a few months ago. Self-hosted email getting dropped for seemingly no reason.


> I can't have it both ways. I would not be on Gmail if it was like yahoo.

I still use Yahoo e-mail for chit-chat (I doubt that it's less secure). Recently it started spam-boxing e-mails from a person I've corresponded with for years (without trouble). Since I have them forwarded to T-Bird, it does mean I have to visit Yahoo more often.

Maybe that's the goal?


> I would argue in this narrow case - I would not point to nefarious undercurrents.

There are so many such 'narrow' cases, and they all just happen to work in Google's favor.


I don't run any spam filters at all, with widely published mail addresses.

I'm getting about 2 spams per day, which doesn't really bother me.


> Google's ML mail spam models are that way because there's too many bad actors. Email, even in 2019, still extremely valuable.

Instead of just blocking bad actors why not work to prevent bad actors from being a problem in the first place?

Oh right, it would go against their business model.


Can you be explicit how they would go about preventing that?


I run an email security/monitoring business [0], and we hear this issue with Gmail a lot, in fact: we have been bitten by it as well during the early days.

The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.

We build our software suite because we found that mistakes in email configuration are easy to make and really hard to identify.

I did a quick scan of rafa.eu.org and found the following:

- SPF is setup [1] with a neutral 'all' mechanism, which basically disables SPF for this domain. Hence, DMARC will set SPF as 'pass' even though SPF did nothing here to help the spam algorithm to assess the sender.

- A DMARC record exists and has a valid syntax [2], but the proposition (p value) is set to 'none', which basically disables DMARC altogether. It will enable reporting, but nothing more than that.

- I don't know you DKIM selector, so I can't assess that, but make sure the DKIM signature address is aligned (using the same domain name).

- The TLS configuration of your email server uses a self-signed certificate and is very much outdated (it offers SSLv3 and RC4 ciphers). I used testlssh.sh [3] to check this.

So yes, you technically did setup TLS, SPF and DMARC, but in all 3 you have configuration errors. Running a mail server in 2019 requires a bit more work and maintenance than it used to be 10 years ago, but the reason is spammers, not Google. Remember: it is in Google's best interest to have as few false positives, as it ultimately benefits the users. It's just really frustrating that it is hard to figure out why Google is marking your email as spam.

[0] https://www.mailhardener.com

[1] https://www.mailhardener.com/tools/spf-validator?domain=rafa...

[2] https://www.mailhardener.com/tools/dmarc-validator?domain=ra...

[3] https://testssl.sh/


No, it's incorrect to say that running an email server requires just "a bit more work". Google basically no longer allows low volume servers to send mail.

I wrote more about this here:

https://www.attejuvonen.fi/dont-send-email-from-your-own-ser...


And if you read the rest of that mailop thread, you'll see that the broader IP reputation of the /24 where your IP is located is a key part of whether or not your email gets delivered.

I would venture to guess that a substantial portion of people reporting issues delivering at small scale are sending from a low-reputation netblock. (Assuming auth is all set up optimally.) So it's not that it's not possible to send at low volume, it's just that most people don't know to look at the netblock, it's hard to evaluate netblock reputation, and it's not always possible to select a preferred netblock within a given provider (especially VPS providers with low-cost options).

The bottom line is: it's hard to run your own mail server, and much of the reason for the success of services like Sendgrid is a) it's easier to set up auth properly with most of them, and b) they actively manage the email reputation of their netblocks (though some do this better than others).

If you're approaching things from a practical perspective instead of making this your hill to die on, you just use an email service provider, and for the vast majority of senders, life is a little bit easier. There's plenty of ESPs to pick from, and there's a very good chance you can find a free one that delivers better than an average VPS or IP of a residential ISP.


My low volume server can send mail very nicely. You can't however do your job half-assed, lack of strong SPF/TLS/DKIM isn't okay from both Google's and anti-spam perspective.


Doing all of that from a popular low-cost hosting provider like Hetzner is still insufficient to be able to send emails to new contacts. You still end up in spam.


How do I choose provider? I checked my IP via talosintelligence.com and it seems that my /24 subnet is mostly neutral with 1 good. Using vultr.


Same here. I also installed https://mailinabox.email on a server and can send email to gmail recipients without trouble. I had some trouble with one of my domains a couple of years ago but it was only related to misconfiguration on my side.


Can you show proof for these claims?


How are you verifying this? I encourage you to try sending to a gmail address you have never sent to before, I virtually guarantee you are going to end up spamboxed.

I know lots of people that run low volume personal type servers, many have claimed that gmail doesn't spambox them, but they were always testing on a (handful of) personal account(s), when I got them to check with an account they had never mailed before, they were spam boxed literally 100% of the time.

Properly setup, and strict FCrDNS, DMARC, DKIM, SPF, TLS and a clean IP and an old domain that have both never sent spam are absolutely not enough to avoid gmail's spambox.


I have verified it by helping people sign up for the small service I run and seeing the e-mails pop in their primary inbox. I guess I might be lucky then shrug.


If you would have actually read the link in the post you're replying to, you would know that Gmail is not accepting mail from me despite that I've spent a huge amount of time in verifying that everything is configured correctly.

Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.


I run a personal mail server for just myself and I’ve been getting full delivery to Gmail so far as well. Same with Outlook.

I’m waiting for the day they try to shut me down.


Can you show any evidence for these claims?


I don't believe you. Can you show proof?


I don't think it's magic, but okay, I don't have a reasonable way without doxing myself to have you test it out yourself.


There's plenty of low effort ways you could provide proof while maintaining relative anonymity (if you actually wanted to support your claims with evidence). For example, you can run a GlockApps test (5 minutes, free) and then take a screenshot of the results with identifying information blanked out.


Because OP talked about Google not allowing small players to send e-mail here's my server's deliverability results to Google: https://i.imgur.com/sceZogI.png

If you're wondering about the blacklist number, that's an error - the source site claims it knows nothing.


Thanks for posting this!


> Also: please stop spreading unsubstantiated claims about your magical server magically sending mail past Gmail's spam filters. There's virtually no chance of that being true, and if it somehow was true, extraordinary claims require extraordinary evidence.

I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim -- I can understand that they often block legitimate mail like in your case, but that doesn't mean that everyone has the same issue. I have occasional issues sending to Gmail but most of the time they accept it and don't mark it spam.


>I'm sure most people who run low-volume mail servers here have no problems sending to Gmail.

Based on direct experience and >testing< (I have talked to about 20 people with personal mail servers about this, and got them to attempt to send me email, on an account that had never interacted with their domain before), 100% of their emails were spam boxed.

Lots of people test gmail with their personal accounts, and think that email to gmail in general is delivered.

In my experience, it basically never is if you are a small sender.


> I'm sure most people who run low-volume mail servers here have no problems sending to Gmail. I don't know why you think this is an extraordinary claim

Where are all these magical people with these magical abilities? How curious that not a single one of them is able to post any kind of evidence to support these incredulous claims. And funny how my comments on HN are getting downvoted and hidden when I ask for proof.


Sounds like there's a niche business and/or OSS project there. Create gmail accounts. Install email send bot on email server. Send messages to Gmail accounts. Figure out how to avoid spam.


Won't this be exploited by spammers?


A ton of services like this exist. For example, Postmark is specialized in sending transactional mail past Gmail's spam filters.


https://testssl.sh/ is super useful, thank you for sharing


Its inaccurate to say that SPF, DMARC and DKIM setup this way is a "misconfiguration", It still allows anybody you are mailing to verify that email is from your mail server, and your domain, which is good enough for anti-spam purposes.

Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.

In my experience it does not matter if you have very strict SPF and DMARC policies, with a good reputation IP address, and an old domain, both of which have never sent spam.

GMail will absolutely still spambox you.


> Setting SPF to neutral (or softfail) and DMARC to none just ensures stuff like mailing lists forwarding mail from your domain aren't automatically marked as spam.

Not exactly. DMARC inspection passes if either SPF or DKIM validation passes. A mailing list forwarding mail will break SPF, but retain the DKIM signature. So you can definitely use strict SPF and DMARC with mailing lists. In fact, retaining mailing list compatibility had priority when the DMARC RFC was drafted. See the RFC.


Lots of mailing lists add footers to the body of a mail, or otherwise mess with the message which is going to cause DKIM to fail.


That's horrifying [0]

An email service that alters the content of an email message in any way should definitely be marked as malicious. This allows for phishing (replacing a URL in the email), censorship, fake news, etc etc.

[0] https://xkcd.com/1172/


> The problem at hand is that it is really hard to debug situations like this. Google won't tell you why a particular email is blocked, because that information will be immediately exploited by spammers.

This is a pretty big fallacy, because exactly this approach affects normal users much more than it affects spammers. The solutions themselves would be much different if you have to pass 10 emails per day with 100% certainty, versus a million where a 5% pass rate is good enough.

A better explanation here is that these denials appear to be somewhat random in nature, driven by multiple layers of AI and proprietary systems that may be difficult to explain to the uninitiated, without revealing the whole code of the algorithm into OSS, and it's not quite possible to explain exactly why the AI does the things that it does.

The only solution would be to actually make it possible to troubleshoot this to an actual resolution. Why it's an issue for a domain owner with a dedicated IP address to send a couple of emails per day from their domain name without ending up in Spam is a bigger question here.

> I did a quick scan of rafa.eu.org and found the following:

All the issues you list are non-issues, and merely the perpetuation of the security theatre.

SPF is very useful for a domain like gmail.com, where there are hundreds of individual IP addresses that may be doing the sending, with little ways of finding out if they're legit without SPF. But the domain of the OP has a single server with a single dedicated IPv4 address. It is trivial to verify whether or not the domain owner has authorised a given email sent from such server, and whether it's authentic, even if it doesn't have DMARC, SPF or even MX records, not to mention DKIM, provided that it gets sent from the A or AAAA address of the domain. Yet it's these very emails that get denied, and even putting these useless and unnecessary for the situation DKIM, SPF and DMARC entries doesn't help, either.

Further, your requirement of configuring SPF and DMARC to reject actual forged mail stems from a belief that someone else is abusing the domain reputation to send forged mail and spam, but, (1), that's never been asserted or implied by the sender, and, (2), why isn't Google smart enough to reject those messages then, instead of rejecting these ones that clearly do come from the owner of the domain, due to the match between reverse DNS, forward DNS, MX, EHLO, MAIL FROM et al. Because that's really what's going on here. We're not even talking about mailing lists or forwarding at this stage; the most basic mail setup results in his messages being rejected.

Here and in your another comment in this thread, you're portraying all this advice as expert advice without even having anecdotal evidence to back it up that it actually works; this is sadly a common practice around the internet, to make suggestions for the sake of making suggestions, wasting people's time on implementing solutions that won't make any difference, where the proposed solution has little correlation to the issue at stake. And this is exactly why you get downvoted on the more technical forums like Hacker News for offering this sort of technical consulting for free — because folks here know better, and because these sort of blanket suggestions don't actually do anything for the situation the user experiences, don't solve the problem, and are technically incorrect and superficial to start with.


Google's making us play by their rules:

1. Our ML models see that your emails spam? You're spam.

2. You're using Tor or a VPN? Here's 10 reCaptcha's.

3. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.

4. Youtube video includes education hacking tutorials? Demonetized.

Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.


>2. You're using Tor or a VPN? Here's 10 reCaptcha's.

I get recapthca ALL the time for just not being logged on and deleting all filthy cookies on exit; and not "honoring" 3rd party cookies at all... and I don't care. Best part is that if you live in a country you don't speak the language you get google services in one of the country official languages -- so what you should do and mark is often unclear. The next best part is a road trip through Europe (edit adding "?hl=en" to the url tends to fix the issue in most of google services)

>4. Youtube video includes education hacking tutorials? Demonetized.

History is a far greater offender.

>1. Our ML models see that your emails spam? You're spam.

Years ago, I recall my company hosted gmail started detecting CEO mails as spam... and he was pissed for not getting responses.


> (edit adding "?hl=en" to the url tends to fix the issue in most of google services)

Seriously, you shouldn't have to. Your browser sends language preferences, doesn't it? Nope, IP trumps user preferences.

Had the same problem when using my job's VPN with an exit point in Germany (working from France). Very annoying.


>Your browser sends language preferences, doesn't it?

Since http1.0[0]... I have wanted to add another header -> "RespectAlHeaders:All-Custom-CrRafted", so the wimpy implementation of ip->country->language is duly removed.

[0]: https://www.ietf.org/rfc/rfc1945.txt


Google also forces certain countries to use Hijri calendar with no option to switch to Gregorian.

Even if your language is English and your calendar is Gregorian, it will still display news dated in Hijri. No setting or option to change.


Ok, that's cruel


I was getting really long and annoying recaptcha all the time, presumably because I use Safari which has all that anti-tracking stuff. But lately (past 2-3 months?) I only get a single challenge, I wonder if they've found a way around the protection or if they've just changed the model


OMG, me too, and all I did was to tighten up cookies (disable 3rd party trackers, erase all cookies on close, etc).

Now I've got a PhD on solving "traffic light" and "crosswalk" Captchas as a result.

I started to resent Google for this, except now I think twice before going to a web site that will give me these hurdles, so it ended up helping me to focus on my work. Now I appreciate the extra work.


>Safari which has all that anti-tracking stuff.

Default Safari option for 3rd party cookies is: "from visited". Check if you have cookies set, if you do - they know you have been a good boy.


It's not that simple with Intelligent Tracking Prevention.


FWIW I've observed the same change in Firefox on Linux (with privacy.resistfingerprinting enabled) in the last few months.


"Moving to Fastmail" is such a popular alternative that if they ever get to any kind of reasonable size we'll be in the same position all over again.

That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.


I have mostly moved to self hosting email. I still have a lot of legacy registrations with various companies that go to gmail, otherwise everything else is self host on a hetzner vps (2.50 eur per month) which doubles as a wireguard vpn.

First time setting up an email server was quite a chore as I have no background in email, getting all of the moving parts to play nice required some patience.

My second email server used Mail-in-a-box scripts which I highly recommend. Zero to functioning email server in a few minutes. The slowest part is DNS propagation. Anyone comfortable setting up a domain name and the basic DNS settings that are invloved will have no problem self hosting email. The setup includes web-based client. I use k-9 mail client on my phone.

I find the catch-all email feature very helpful in creating ad-hoc addresses for spammy companies (eg car insurance). I can sign-up with spammy.companyname@mydomain.com and it will direct to spammy@mydomain.com.

I'm not completely de-googled, but I am consciously and gradually reducing my exposure to google, and other large networks.

At the very least it is worth a try.

https://mailinabox.email

edit:spelling, added url.


I've been using Fastmail for about 5 or 6 years now. My only problem is the 1GB mailbox size limit I have. Maybe I should upgrade. Otherwise I love it. Their webmail client is phenomenal and works with mobile and desktop. I don't like their Android app but I haven't tried it in a year. Their web client is so good I don't need an app for them.

I do see mail going to spam though. It's never bugged me that much.


You can also just use the native imap client provided by your operating system. The beauty of using a provider and service that adheres to open email standards.


We're still a fair way away from that! We're growing steadily, but not at the same rate that "give it away free and work out how to monetize it later" services do. Worth it for having a sound business model and no venture capital breathing down our necks to turn evil though.


Thank you for not being evil.

Please allow me to export notes to txt.


hah - nice segue to a feature request! You can access notes via IMAP, so that's an easy way to get the whole lot raw.

As for a more general download - what would you be looking for? A zip file containing all the notes?


Thanks! I was looking for an export button for them earlier today so when I saw you comment I couldn't help myself.

A zip file with all notes in it would be fantastic. I just want to feel like my data is safe from Apple having a bug that wipes it. Read a lot of Catalina horror stories.

IMAP looks promising and might be all I need. Neither Spark or Apple Mail seem to let me export mail as txt but presumably there's a mail client out there that does in which case problem solved. Thank you for point that feature out, should throw it up on:

https://www.fastmail.com/help/account/downloaddata.html#tran...


It's not the ideal solution and certainly not as easy as clicking a button but, for a quick 'n dirty solution, you might look into using "offlineimap" and configuring it to sync only your notes folder instead of all of them.


You're right.

I use Fastmail scared that my emails will go to spam. But ideally I want to live in a world where my emails hosted on my email server will send like they did 10-20 years ago.


I keep using my own email server to host my emails but had to switch to fastmail for smtp since it was untenable otherwise...

It's sad that it's no longer viable to have your own email server


I want to use Fastmail, but their minimum price for using your own domain is $50/year. At that price I can get Exchange Online with double the storage. And given that they have an office in the US, I would expect them to follow the same laws that Microsoft and Google have to regarding requests from government agencies.


This looks to be a good alternative except it does not have HIPAA support.

Even though I have shut down my practice for the near future, I will still need to have information protected for several years to the Health Department's requirements for psychological records.

Email providers who lack this support will have difficulty gaining traction in the health world.


You shouldn't be using email to send PHI at all.


But fastmail lets you use your own domain, so there's much less vender lock-in.


The problem is that smaller companies may not have an appropriate budget for security.


> Before they become a huge conglomerate which controls major aspects of many people's lives

That's already the case.


Could you sue Google for slander in a case like this? They are explicitly labeling your messages as being manipulative or dishonest without any just cause. Why does it matter it’s an algorithm instead of a careless person in power?


For those of us that can, we must vote with our dollars. To divorce ourselves from Google is expensive and painful, but to ignore it is catastrophe


but it's not expensive. there are other free email services, or you can pay for decent email hosting for a few bucks a month.


Could you post examples?

In my opinion free email services either belong to other big corporations or provide lower quality service than gmail.

Few bucks a month add up and can be considered expensive by many.

I don’t like google and I pay for a different email account. I just wanted to remind that gmail may be hard to quit and that costs are subjective.


I have all my mailboxes with gandi.net. Each domain you buy from them comes with 2 fully managed emails, and I think 3GB storage each, for just over $1/month. I basically never get spam, nor blocked emails. They have 2 webmail interfaces too, but I don't use them. As a bonus, gandi financially supports a ton of open source projects, which never hurts.


I use https://mailbox.org/en/, it's from a small German company and I can use my own domain with the smallest plan for just 1 € per month. Their spam filter also seems pretty good.


Until the day you get german spam mails because all spam filter are trained for english mails. Had to add a few keywords to tune the spam filter.


> Few bucks a month add up and can be considered expensive by many.

I pay around 3$/month for Fastmail, I don't think it's expensive, an email address is the primary communication method everyone uses nowadays and it's useful every day.


Protonmail is my choice, it supports client-side encryption. Custom domain $5 a month - unlike Google you are not paying with your private data.

https://protonmail.com



If you're not paranoid about Russia, you might consider Yandex Connect. It supports custom domains and has free tier with ads in the web interface.


Goog luck with that, Gmail is free.

We could probably start to divert all gmail and outlook originating mail to the spam folders on our mail servers in retaliation.


> but we need Google to stop this "my way or you don't exist" attitude

I think that's exactly opposite. We need more competition, not forcing a private company to behave in some way that "we" (and who is "we"?) find acceptable.

It's time to break them up.


Why is it necessary to break them up when the alternatives are right there?

Google can exist, and people can choose to use Bing, DDG, Fastmail, etc., right?


That's a very fair point.

The problem is that Google is not just Google the search engine, or Google the email provider, or Google the video service, or Google the website analytics engine, or Google the online office suite, or Google the ads syndicate.

Because Google is all of those things, it has accumulated too much market power, to the extent that they are market distorting. Bing and DDG (which is not an independent engine, but collates results) are irrelevant to a company's Internet presence. There is no such thing as SEO for Bing. Whereas Google can make-or-break you.

When Google the ad syndicate and Google the search engine collude (as it were) to place your competitor's ads ahead of your organic search result, a top result even, and to distinguish such ad results as ads only technically, by the narrowest of definitions, this compels you to advertise with Google the ad syndicate. Just as an example.

This behavior is harmful to the market (not to mention to users, but that's a different subject) and it is a given that Google the monopoly^Wsingle entity will continue to make such moves, whack-a-mole style so it's futile to squash any individual behavior even if you could. Ergo the reasonable solution is to break them up.


Have you actually had a lot of problems sending e-mail to Gmail users from your mail server? For me Gmail has not been great about accepting e-mail and not marking it spam (occasionally have issues), but others like icloud.com have been worse.


>Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.

This ship sailed years ago, though.


I mean, I get captcha's all the time while working from Google's network. You're exhausting from a shared quota. That's life on the modern internet.

Bitch about Google all you want, it's not like they decided this is how the internet should work. Prior to Google, my company integrated with reCaptcha because no other choice existed to reduce automated abuse.


FWIW, I tried creating a Hacker News account via Tor. I was hit with an unending stream of CAPTCHAs. Eventually gave up.

I was surprised/disappointed to see this happen on HN.


Possibly because HN was under attack a few weeks ago. More about that here: https://sdan.xyz/hnhack


This was months or years ago.


Ironically most spam I've been getting recently is from Gmail addresses.


> before they become a huge conglomerate

too late?


> Google's making us play by their rules

Yeah well, they offer a free service and you use it. They own their network and they can decide what they allow.


[flagged]


What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things.

It seems you're not getting the point. The problem is not for users of those "awesome" Google services the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account.

Google tags those as spam and refuses to deliver them, while always shifting the goal posts and providing sub par support (an number of essentially useless links) to the sender of the email to a Google account.

This happens, btw, also when the sender account never spammed and sent legitimate mails to the Gmail user for years.

So your snarky comment is completely out of line here.

It's not about people mooching free "awesome" Google services for free it's that Google is a rotten net citizen and is behaving like shit to the entire world.


> the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account

It's funny to me that people in this thread are complaining about Google being large enough to serve as a de-facto authority, when the only way one could solve a "legitimate emails are being blocked" problem is to have a definition of 'legitimate' that would come from... an authority.

Google is dropping the ball but I don't know what the solution would look like that isn't just "Someone else be Google now." ;)


What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things. It’s so demoralizing. Get off your high horse.

I don't use gmail these days (except as a honeypot out of laziness), but I get plenty of email from google hosted domains. The issue isn't (just) that people want to use these services and want Google to have more reasonable policies, there is also the issue of having to tolerate Google's policies because other people use their services.

Emails from Google servers are notable to me because there is no way to report spam (and the signal to noise ratio from Google is pretty low for my case). Google's opaque approach to email means that deliverability issues become even more tedious to deal with than they should be.


It’s been a while since google made anything awesome hasn’t it? Maps is still great, but things like AMPs are so terrible that they were a bigger part in driving me to DDG than privacy.

Maybe I’m weird, but I’ve never been in an AMP where I wouldn’t rather go to the actual side and use reader mode.


Isn't duck duck just redesigned yandex (aka kgb) search?

At start it has mark that search results based on results from yandex engine and crawled results


Ddg is a wrapper around Bing. They may incorporate other results as well.


For free ofc


I'm actually having a similar problem recently, although my messages are declined at the SMTP stage, "due to the very low reputation of the sending domain", which I attribute to the output of certain cron jobs being sent towards my own Gmail account — it seems like my whole primary domain name has subsequently been blacklisted a few weeks ago:

* https://news.ycombinator.com/item?id=21340460

* https://mailman.nanog.org/pipermail/nanog/2019-October/10381...

It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.

Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.


I have a hotmail throwaway account (it has since been migrated to outlook with the microsoft buyout) and its spam filter is completely backward. There is currently 373 e-mails in my inbox (100% spam, expected since it is a throwaway...) and 4 e-mails in my spam folder--100% of them the account registration/confirmation e-mails that I was wanted.

I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.

In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...


> it has since been migrated to outlook with the microsoft buyout

Aside: Microsoft bought Hotmail in 1997! How old is that account?!


Honestly not sure, but certainly sometime in the 20th century. It was my main e-mail back then, I migrated to Yahoo! shortly after the Microsoft buyout and kept the HoTMaiL as a throwaway. When Google announced Gmail and the 1GB (with projection to infinite) storage I switched to them as soon as I got an invite, but when they changed their slogan from "Don't be evil" to "Do the right thing" with the Alphabet restructure I ditched it and moved to my own domain with Fastmail. If Fastmail starts to show signs of evil it will be as easy as updating my domain records to point toward OurMailIsActuallyGoodIPromiseThisTime.com and click through the certificate warnings on my mail clients.


That Hotmail branding survived long after the purchase; in the last few years it's the same interface as their 365 business offering now, and under outlook.com branding.


When I last had to wrangle with MS (c.5 years ago, who were dropping all our email _replies_ to customers, they had a third party that you could pay to ensure your emails got through ... apparently the problem was our domain was on an IP hosted by a company who'd previously hosted (but didn't now) a company on a different IP and that other company has sent some spam.


I sometimes wonder if certain companies can get white listed.

There's a particular perfume company that sends spam to my gmail account every month. I've been marking it spam for at least five years, but somehow it always gets through.

Remarkably, Fastmail has gotten to the point where its spam filter works better than Google's for the kind of messages I get.


The interesting thing about whitelisting is that you have to be sending a lot of emails so the company determine if you are sending spam or not. So if you have a private domain, your own mail server, and are sending only a small number of emails then your not going to get whitelisted. Sometimes you can get on a feedback loop but all that will tell you is if someone clicked on "this is spam".


In testing different mail servers I find that Gmail will treat them differently based upon there hello line. I can't remember which one was accepted most but there is a difference. Also you need to look into if others on your netblock are sending spam. If you are on the same netblock as a spammer then gmail will treat you as if you are the spammer.

This site is very useful if you have your own mail server https://www.senderscore.org/


Heh, I get a recapcha from that site. The irony.


The biggest problem to me with Gmails spam filter is that it is so totally hit and miss, and there seems to be no reliable way of ensuring that messages stop going to spam, even from other Gmail accounts, from people on my contact list, that I've persistently marked not spam.

At this point the only reason I'm still using Gmail is sheer inertia, and it's a question of when I get time to start untangling myself from it.


Have you tried filter rules?


Yes. I've also used Google Apps and had it file e-mails from our own users as spam despite settings that's supposed to bypass it.

Meanwhile some Gmail addresses I have rarely have this problem, so it's not even consistent across accounts or domains.

The most annoying bit (and probably the first untangling from Gmail I'll be doing) is that I have my own domain hosted on Google Apps, but since my main Gmail mailbox is a different one, it's set to forward everything. It's meant to bypass all filtering and just forward. Or so the settings and support claims. It doesn't. I have to regularly log in for the sole purpose of marking things not spam.


Not sure about OP but to me rules in GMail don't work.

Gmail filters don't seem to work anymore. I had filters and labels and it was great but I noticed they seems to be failing.

If you go down the rabbit hole of Google Gmail support you'll find filters don't apply to email ("conversations") already in your inbox. But they also don't seem to apply to new email.

Asking on various groups people seem to be giving instructions for options in GMail that no longer exist.


I think they change the way it works, and then existing filters may break at one point. I've had this happen to several filters.

TBH, it's just easier to re-do the filters. There's also an option to make it apply to old conversations on a one-time basis as well.


I've tried that without success. Delete all rules/filters, labels and re-create it all again. Nothing will work on old or new messages. And "apply to zero conversations" is the only thing I see that seems like it would be it but the zero part means no chance of working.

Gmail even replied to me on Twitter saying the filters no longer work.

Screenshot of Gmail reply on twitter https://i.imgur.com/URhBeXn.png

I like using filters I don't see why such an important email tool is so awkward to use on Gmail.


It doesn't seem to work. In fact, IIRC, they simply add a huge button dialogue to any such emails that pass due to a filter to mark it as spam, without any buttons to dismiss their improper suggestion — e.g., you can't even confirm that the mail isn't spam anymore if you use such filters.


If the mailing list is using a bulk email service, then they should use DomainKeys Identified Mail (DKIM) and Sender Policy Framework (SPF) for identifying ownership & not getting marked as spam i.e. If they aren't really a spam.


Sorry, but your advice have been detached from reality for a long time.

SPF is not a "should" anymore, it is a must. Many major e-mail providers will outright reject your e-mail at SMTP level, unless you have correct SPF record. Rest will move SPF-less e-mail to spam on automatic basis. In this sense SPF does help you to get delivered, and avoid getting into spam bin, but your chances can only go from "100% spam" to "90% likely spam", not "0% likehood of being spam".

DKIM is not a major consideration for Gmail (or anyone, as far as I know) and have never been one. You will not magically avoid spam bin by using DKIM. Instead DKIM provides a great opportunity to botch your reputation with mail servers: as soon as you forward a e-mail (between providers or within the same provider) it will lose it's DKIM validity, and thus may get classified as spam (!!) on the basis of invalid DKIM signature. Boom! you are a spammer now.

Valid SSL cert and valid SPF are the only things, that have noticeable impact on Gmail's treatment of your email. DKIM and DMARK and largely useless and can only get you in extra trouble.

As for bulk email, it is going to be classified as spam until your domain and IP gain sufficient reputation. The only way to gain that reputation is to send slowly increasing amount of email for long time without being marked as spam. Being a rich business, Google's client, hanging on Google's forums, using Gmail as your mail server and having at least dozen contacts among Google's employees also helps.


This same thing started a few weeks ago for me as well. I have a cron job that parses my log file and sends me usage statistics every morning. I had been receiving this email for 100+ days when suddenly it stopped. I investigated thinking it was a problem in my script only to find that all worked well and all emails were in my spam.

I use SES from AWS and now my new user sign ups emails for email verification also end up in spam. It’s very frustrating because they changed some rule and now my low volume email sending is considered spam


You know you can set a rule to send nothing to spam? Or to whitelist certain senders/domains/subjects?


I thought this had been getting better. At least more e-mail from my personal server has been getting through to its destinations. I've written about this too a few years back:

https://battlepenguin.com/tech/how-google-and-microsoft-made...

..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).

I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.

My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.

But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).

Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.


> but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.

This is why I like that my email provider (https://mailbox.org/en/) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.


but what if email is the only way to reach you?

so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.

depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)

if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.

the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.


> but what if email is the only way to reach you?

I have no idea if it actually is, but it might be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.


Wouldn’t a simple contact form on a website meet the definition of quick and electronic communication?


Yes, but that has it's own challenges with spam protection, privacy laws, users not spelling their email correctly etc, and you still need an email address to receive those messages (or check for new form entries regularly), so an email address is often used as a shortcut.


I share the sentiment and the frustration.

That said, if you read far down enough in the thread, it's pretty clear that the OP has very, very unrealistic expectations about others' obligations to him, and very incorrect notions about how statistics work. The argument eventually turns to "well google has a lot of processing power at its disposal so I don't see why it can't use this in order to turn incomplete data into more complete data" and that's when I checked out :\

Edit: also this Brandon Long has the patience of a saint.


The attitude of the replies he gets lower down in the thread is shocking. I hope whatever mailing list that is doesn't have any kind of good reputation, because it seems like the consensus opinion there is shockingly anti-cooperative, anti-internet, "me before the network", even going as far to say "if Google thinks you're a spammer then you probably are".

How things have changed.


It’s posted elsewhere in this comment thread but that’s after the point where it becomes clear that he wrapped a moral argument in a technical envelope and people spent a lot of time trying to help him out with the technical argument.

Put another way, play stupid games, win stupid prizes.


That's a meeting of technical and moral view. I'd love for corps to have no impact on people, for distributed knowledge and resources, for everyone being in control of their data, and general socialist direction.

But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.


> Posting about how unfair that is is not going to change anything.

It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:

"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - and that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."


Might be a good time to mention a new development with Yahoo Mail. They now appear to use a form of shadow ban.

They will accept an email, with all outward symptoms being normal, but it will not appear anywhere in the user's mailbox. Not even in the Spam folder.

This started few months ago when we were sending out our (rather infrequent) newsletter. 5th or so message to ...@yahoo.com address failed with

    421 4.7.0 [TSS04] Messages from xx.xx.xx.xx temporarily
          deferred due to user complaints - 4.16.55.1; see
          https://help.yahoo.com/kb/postmaster/SLN3434.html 
          (in reply to MAIL FROM command)
That's with just a couple of seconds in. They kept replying with 421 for a while, but later switched to just dropping connections. Then, in the course of the next few days, they would accept 1-2 emails and then go back to denying them again.

As I mentioned above, the "best" part is that none of these emails would actually make it into recipient's mailboxes. They disappear.

Better yet, all subsequent (non-bulk) emails have the same fate as do _replies_ to emails from Ymail users. This last bit is what makes this look like a shadow ban.

All attempts to contact Yahoo went unanswered, including those made through Gmail. Ditto for whitelisting requests through their web form.

If anyone has any ideas what the hell's going on, I'd be curious to hear them.


I don't think this is new. Many email providers will throttle / blackhole you if they decide you're sending too fast or see too many bounces.

Slow down your send rate, make sure your server is configured correctly, and prune your lists.


"Many providers" - any concrete examples? For an SPF/DKIM'd server that's been at the same IP for nearly a decade and had not a single spamming incident.

PS. The list is immaculately maintained, the server is configured as properly as it gets and very conservatively rate-throttled. There's nothing obvious.


Gmail also occasionally blackholes mail.


Gmail's ML models categorize plenty of Google's own emails as spam.

I honestly don't think they are nefarious, just overly insistent.

I remember email before Gmail's spam filters. Everyday was a task of deleting spam first then responding to emails.


This is the consequence of the spam filters doing a good job. I too remember when consumers would clamour for better spam detections and different companies would hawk products that purported to do so.

But now that spam detection mostly works well for most people, it’s a forgotten issue. Now we have people on this thread questioning Google’s authority to filter spam because from their PoV, Google isn’t adding much value to begin with.


The problem is that Google errs too much on the it-is-spam side. Having to delete a few spam mails isn't a big issue if the alternative is losing legitimate mails because of overzealous filters.


That may be true for us, tech savvy users.

Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.

It's a very hard problem to solve.


Sandstorm should have really caught on. We need to make self hosting drop dead simple so that even non-technical people can do it. The power of these mega-tech companies can only be whittled down if we take collective, well thought out action. And it is not just about tackling power imbalances. Owning your own data is a fundamental aspect of our digital lives which ought to have become simple and natural, given Moore's Law and the phenomenal device capabilities in recent years, yet somehow we are going the other way. Even as our devices become ever more capable (hardware wise), the software is moving to ever more to the Cloud and we're being locked out of the full capacities of our devices except in the narrow realm of media consumption.


Since people specialize, we have organized societies and legal systems so that people can use services with reasonable trust.

With new technology, the laws of course lag. You see a wild west, then monopolies etc before it settles.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: