More directly: spammers are killing independent email. Email's peer-node-trust story is so "version 1.0 Internet" that webmasters are left basically using heuristsics, shared models, and tea-leaves to determine whether arbitrary incoming messages should be trustworthy or not, and "they should not" is a good first-pass guess!
So Google (as the thousand-pound gorilla) is serving as a lightning-rod for a larger network-effect problem, which is "Users generally consider themselves better served if most unsolicited email they receive with no strong trust priors drops into a black hole." But that makes it very hard to be a newcomer who wants to establish trust priors.
> More directly: spammers are killing independent email.
Disagree.
In the early 00s, spam was killing email. I was getting thousands of spams per minute to my personal email domain (same one I still have and have had since the mid 90s). It was real bad back then.
Doing aggressive sanity checking on incoming email back then was problematic since nearly all legitimate senders also has badly misconfigured email servers.
Those days are gone. Legitimate senders have well-configured email now, one can do a lot of sanity checking on the SMTP connection and that alone cuts out most of the would-be spam before it is ever sent and none of the real email is impacted.
Sprinkle a bit of bayesian filtering on what remains and spam is mostly a non-issue anymore.
So what it the problem these days? Part of it is a legacy cultural problem. A lot of email admins wear so many scars from the early 00s that they still operate on a mental model that they are willing to throw away a lot of legitimate email just to prevent even one in a million spam to go through. That's highly counterproductive and causes a lot of harm to interoperability.
The other big factor of course is that the very deep pocket email providers (gmail, microsoft) have a strong vested interest in strenghtening their monopoly even further so they are happy to go along anything that decreases interoperability.
That all said, I strongly encourage everyone who can to run their own email. Interoperability can only be saved by exercising it as much as possible. And it still works fine despite the naysayers. I've been running my email infrastructure for a long time and have no deliverability problems anywhere that I've run into.
Well then they for sure can't use gmail. If you still do use gmail, please consider moving to something else. And until you actually do, make sure to keep a close eye on the spam folder because you've most probably lost important mails.
I couldn't disagree more. I have several email accounts and the gmail one by FAR lets through the most spam, a lot more than my Fastmail emails which despite being more published have far less spam than Google.
But do you know for a fact that it lets through a larger proportion of the spam sent to it, or are you just noticing that it gets a larger total volume of spam than the others? Because if it's the latter, that could be an effect of more spam being sent to your gmail than to your other accounts.
> That all said, I strongly encourage everyone who can to run their own email. Interoperability can only be saved by exercising it as much as possible.
While I have no doubt that doing this is a lot easier and more hands-off than the last time I did this (for a few years during the mid '00s), I just don't have the time or energy for it. Also, email is a critical service, something where I don't want to have to deal with downtime, especially if I'm away traveling or something like that.
I think as a decent alternative for people like myself, it's at least helpful to use an alternative hosted email provider (other than GMail or Microsoft) to help keep the ecosystem open.
> Also, email is a critical service, something where I don't want to have to deal with downtime, especially if I'm away traveling or something like that.
I have to say that as the most critical service of all, that's one of the reasons I do host email myself!
I can't and won't take the risk of being locked out by the whims of some third party misconfiguration (we've all surely seen the endless stream of misfortunes by those getting locked out of gmail/etc).
I assume badly misconfigured servers meant something like open relay setup. Not DMARC and dependent technologies, which are not protection against spam at all, anyway. They are protection against various kinds of spoofing.
And I concur with the original commenter. I do zero filtering on my SMTP server (no reverse DNS lookups, no blacklists, no nothing), in order to be sure I receive every single email, and SPAM looks like it would not be an issue even if I disabled the bayes classification client side, these days.
I mostly rely on providing unique random addresses to each service. My oldest 18 year old address gets quite a bit of spam so I blocked it and switched to different one, but newer addresses that I use for the last 5 years didn't accumulate much spam, despite being public and crawlable on the web.
IMO email is dead and spam killed it. Who uses email anymore? It’s basically business communication (but all the biz communication I’ve done in the past years has been through telegram / signal / whatsapp / etc.) and registering to websites.
Once we figure out a way to handle website registration that’s not tied to email (whatever that means) I’m not sure email will be useful for anything.
Most useful conversations have moved to messaging apps.
Email is incredibly heavily relied upon and with great success. People get their pay receipts, their rental applications, notifications of mentions on social media, speeding tickets, music and gig notifications, reminders to get their pet to the vet, X-ray reports, prescription reminders and I could go on.
Not to mention the huge boom in email digests and blogs over the past 5+ years.
I haven't seen - _any_ businesses communication conducted over signal or WhatsApp - ever. I'm not saying it doesn't happen somewhere but it's certainly not common place. And telegram? I didn't even realise that still existed!
> Most useful conversations have moved to messaging apps.
That's not a future I'd like to ever live in.
Why? Because all of those are proprietary solutions that don't interoperate with anything and their behavior and existence depend on the whims of a single corporation. No thanks.
Email is standard, interoperable, owned by nobody, accessible to everyone. It's the optimal way to communicate.
So many people completely miss this point. If whatever is supposed to replace email it will have to be very similar in features. That is
1. Permissionless (open for anyone to send messages or operate a server)
2. Asynchronous
3. Resilient
4. Searchable
No IM solutions I know can do that at all. Matrix would come closest but asynchronicity and resilience are still troublesome on that part and all clients are more geared towards IM instead of async longform discourse
While one could heavily overgeneralize and say that "everyone uses google for mail" (iirc they own like 20% of the email market) they certainly don't own the protocol or its development.
I think it is exactly these sentiments that do most harm.
I mean, that is completely false at every level (technical and operational).
Sure, gmail is a big player but there are many others. Most importantly, there are thousands of smaller players. All of which interoperate.
I can't tell if you say that sarcastically or from a position on being uninformed about how email works. If the latter, do take some time to understand email infrastructure at a high level because it is a case study in beautiful interoperability of open standards. Something we must all keep in mind for the future of the Internet.
Email has been resilient and an open communication mechanism from everyone since the 70s precisely because it cannot be owned by anyone, it is an open interoperable standard.
There will never be anything equally long-lived and feature rich from a single corporation. Corporate interest don't align with that. Proprietary solutions will always be about locking you in and limiting interoperability.
Nome of the service you mention have the async nature of email.
If I want you to be aware of a long article I'll send it to you in an email because if I message it, most people will open it, see it takes more than 2 minutes to read and close it and never go back to it.
Really? WhatsApp is end-to-end encrypted unlike something like email or slack. If your company doesn't understand that they should hire a security team.
I run my own email server in part so I could implement a spam filtering strategy that had access to my outgoing mail. Anything received from someone I sent mail to is assumed non-spam, and that is the anchor for the rest of the filtering.
I was planning to build a pretty sophisticated bayesian filter on top of that but it turned out to be unnecessary. The strategy I ended up with, which has served me well for many years now, is:
1. Block a small number of extremely spammy TLDs. There are about a dozen of these, including .biz, .casa, etc.
2. Anything received from someone I've sent mail to, or a sender I've previously marked as good, is assumed good.
3. Anything received from an address whose name contains an English word on a relatively short list of spammy words (discount, offer, etc.) is assumed spam.
4. Anything received from an address that I have received email from before and not marked as good is assumed spam.
That leaves emails from addresses that are sending me email for the first time. These are overwhelmingly spam, but after the four filters above there are few enough of these that I just scan them manually once a day or so.
The real key here is treating the first email from any given address as essentially a "contact request" with the default action being "deny in the future". That just turns out empirically to work incredibly well. More than 90% of my spam is from repeat offenders.
Anything that reaches addresses that you don't use gets marked as spam.
Common prefixes will work for cases where they're guessing your address, and hidden mailto links will cover situations where they're scraping your webpage.
>1. Block a small number of extremely spammy TLDs. There are about a dozen of these, including .biz, .casa, etc.
Honest question: Is there any reason to not just blacklist all emails from TLDs that aren't the Big Four (.com, .net, .org, .gov) and country TLDs (eg: .co.uk, .jp, etc.)?
I can't recall ever needing, let alone wanting, email from the later TLDs like .biz or any of the bloody stupid new TLDs that comprise entire words.
And I get a fair bit of ham from other weird TLDs. So I think a whitelist is a bad idea. But that's because I prefer to have some spam get through than have ham get blocked by mistake. Even with my very sloppy algorithm, the only time spam ends up in my inbox is if a spammer spoofs one of my contacts, and that is extremely rare.
.co.uk isn't a country TLD, .uk is. If you block .ac.uk you'll block every UK university, .gov.uk the UK government and if you block .me.uk you'll block, well, me.
as someone that’s never needed SEO perks of a .com since none of my customers would use a search engine to find me (they’ll click thru from posts and ads on social media and group chats)
I typically register both a new TLD and a .com, and I use the .com for the email campaigns and reply-to
if it gets flagged that doesnt affect my more official one-to-one emails from the actual domain
so I would say, maybe? because I’m not allergic to “bloody stupid” TLD’s and have already adapted to what you’re wondering about
There are lots of businesses using `.io` `.ee` `.co` etc that depending on your email use-case would be detrimental to block especially if you're doing any kinds of business or outreach.
> Honest question: Is there any reason to not just blacklist all emails from TLDs that aren't the Big Four (.com, .net, .org, .gov) and country TLDs (eg: .co.uk, .jp, etc.)?
> I have no idea how blocking TLDs of entire countries with hundreds of millions of citizens can be considered a valid anti-spam advice.
It is very different if you offer it as a service to other people, or if you do it for yourself.
If you are doing it for yourself you know if you expect emails from country X or not. And if things change and suddenly you have a pen pal from there you can change your filters easily. So you can live with such simplifying assumptions.
If you are providing email services to masses of other people then you can’t make these assumptions of course. You have to use more sophisticated means then.
> If you are providing email services to masses of other people then you can’t make these assumptions of course. You have to use more sophisticated means then.
These things just get thrown in the large pot of heuristics.
If I ran my own, personal email server, I could probably block the TLDs of every country in Asia, Africa, and South America and never actually block a single email. I don’t receive emails from those TLDs and I quite possibly never will. If I expected to, I wouldn’t block it. I don’t travel the world, making friends in every country I go to. That sounds nice, but that’s not my life.
The poster you’re replying to is talking about their filtering method for their own email. Presumably they know if they expect to ever receive an email from a given country.
Is there a particular reason to go ahead and preemptively blacklist every country's TLD you doubt you'd ever receive an email from? Would it not be better like if spam isn't expected to be an issue you might as well leave it open, then if you do one day receive a spam from that TLD, block it then?
Otherwise, let's hypothetically say you have a friend named "jacob peter" who really likes his initials (and doesn't understand the topic of this discussion), so he registers somewhere in Japan that lets him have dot-JP as his TLD. The only hypothetical way he knows to contact you is via your email server (let's also assume he doesn't do stuff like social media, phones, etc., so he can't find any other way and you have no way to contact him). If you have hypothetically preemptively blacklisted all of Asia's TLDs, then you'd never talk to him again for as long as you live, right?
If this entirely fictional friend only ever emailed me from his .jp email address and never contacted me any other way, I guess so, I would never talk to him again.
Since he’s not real I don’t know if that’s a good thing or a bad thing. Maybe he’s kind of a dick.
I used to do this at one time. I had no users who corresponded with addresses in Russia, Romania or China, and those represented a very high proportion of inbound spam.
Nowadays (irony of ironies!) most inbound spam is from gmail.
Not bigotry at all. If my server gets a lot of chinese spam, and none of my users reads chinese, then it's just silly to waste bandwidth and storage on messages that aren't going to be read.
But you aren't blocking the Big5 character set, you've determined that 1 billion people can't possibly learn English and have no reason to contact your users. Because reasons.
When Chinese and Russian ISPs start taking abuse complaints seriously then they can be given the benefit of the doubt. But until that happens they're going to remain on most RBLs for a good reason.
So you've determined an entire population has a certain behavior because of past interactions with other entities from that continent-sized part of the world.
Do tell me more about this wonderful new method of spam filtering with absolutely no bigotry.
Judging someone by their current behaviour is not bigotry. As I said, they have the option of taking abuse complaints seriously, but they haven't for a really long time now.
there's no such thing as a "woke" spam filter... i mean, it'd hafta be inclusive or whatever to make sure none of the email feels marginalized or sad... so to be "PC" it's gotta let everything through... which means it isn't filtering spam...
This isn't about being "woke", whatever that means. It's just about not being a bigot and classify broad people in a certain way because of their geographic position and - by extension, ethnic and national origin.
Just imagine the racket if Gmail would start de-prioritizing email from Alaska or Hawaii because, in their experience, that's a large source of spam. If might even be true, so what? That's the exact thing racists say, racism works: we found the guy suspicious because no black people live around these parts, so we knew he was up to no good.
let me make sure I understand... filtering spam from geographical sources known to produce a disproportionate amount of spam, that someone never gets legit email from as they have no interaction with anyone there, means they're a bigoted racist... so instead we should just not filter spam, it's not fair to the spam for us to microaggressively generalize it into some group that gets marginalized, every email deserves equal treatment regardless of where it originated from (since that's all PC and woke, etc.), which ultimately makes sunshine and rainbows go everywhere and world peace and stuff?
you're making comparisons here that are so disjointed it's not even apples and oranges... it's more like apples and rocks...
this thread grew outta the spam filter setup of someone hosting their own email server. bringing gmail into this and talking about how they'd have a PR nightmare if they screw with hawaii doesn't work. yeah, they both send/receive email, that doesn't mean they're comparable in the way you're tryna say... i mean, apples and rocks are both round sometimes, but you wouldn't try comparing how they taste, right?
This thread grew out of someone claiming blocking entire countries is a good anti-spam technique because they just "know" those people have nothing to say to their users. This is bigotry pure and simple, no need to go any further, they hold strong opinions that can't be objectively verified about large parts of the human population based on national origin.
As someone who was on the receiving end of the "Romanian users have nothing to say to my users and postmastars there can't even RFC" blackholes, trying to get legitimate email delivered while respecting all the rules of the internet, I can only say that whomever thinks like that is an idiot and a bigot. Unfortunately, there are many such idiots on the internet at large, and the aggregate effect is that only large sites like Gmail can maintain good deliverability, which was the subject all along.
Oh, right, I actually forgot that .tv was actually a country code and not a TLD for television.
So... my apologies to the 12,000 residents of Tuvalu. I'm sorry to have to block you all, but for some reason your country code attracts spammers like flies to honey.
Correct me if I'm wrong, but I think .tv domains are pretty pricey. They're loved by people who need a bulletproof domain for their sketchy business but I don't see how they would be useful as a disposable spam source that's going to get burned quickly anyways.
Keep in mind we're talking about a single person here. It's possible that everyone's individual list of tricks would be different given where they live and who they communicate with. If you were to use this blunt instrument on a few million subscribers you would undoubtably experience more false positives than OP experiences.
Honestly, just block all everything except the traditional TLDs like .com, .net, etc. and country-code TLDs for countries you expect to interact with. It's a trivially easy way to greatly reduce the amount of junkmail you receive, and saves on compute resources too (since you can reject mail before it even hits your AV/anti-spam/etc. stack).
In many years of doing this at work I think we've had to manually whitelist three domains (our corporate travel provider started sending from a .travel domain instead of their .com without telling us they would make that change...). I don't have numbers handy but it significantly reduced our junkmail intake.
Isn't that a bit like trying to predict who will want to talk with you? Maybe you are not in that sort of business but I constantly give out my email to people I meet at conferences and through that get Ham from pretty much any existing TLD, how do you cater for this case?
We mainly deal with governments and other businesses in the same fairly small industry, so it's somewhat easier for us in that respect - we do know most of the entities who walk to talk to us in advance. Also, we have a good relationship with our staff and they're usually proactive about telling us when they're expecting mail coming in / we add a new supplier / etc. so we can keep an eye on mail delivery.
There's always some mail that gets blocked, of course, but it's a relatively rare occurrence.
Maybe in your case something that allows users to view and release their own quarantined mail [1] coupled with a healthy dose of training on spotting malicious emails and so on?
In my experience, it changes all the time. A strong factor is NameCheap doing dollar sales on some TLD; causing a surge in DKIM-signed spam using that TLD for a few months.
I run my own mail server too. Rspamd does a reasonable job of cutting back the amount of down that gets though, but I feel like I have no control over it or visibility into what it's doing.
What tools are you using to implement the controls you describe above?
I rolled my own. It consists of Postfix, Dovecot, a custom milter written in Python, some small sieve scripts, and the spam filter written in Clozure Common Lisp using SQLite3 as the database.
Spam is killing tons of services beyond just Email.
YouTube, Twitter, Facebook, Craigslist, ...
Nearly anything public is bombarded with spam. The big players are far better suited to deal with it than any newcomers, and even they can barely manage it on their own platforms.
And it's not even limited to online or free. My snail mail is 90% spam even though people have to pay for it to be printed and delivered physically to my door. I wish I had spam filtering for USPS.
The question I have is, why does the USPS provide a discount for bulk mail at all? Sending bulk mail is less than half the price if a regular letter. It’s basically a subsidized door-to-door garbage delivery. My recycling bin is right next to my mailbox, and my daily routine is removing the garbage from my mailbox and putting it in the bin. Such a ridiculous waste. And it’s my responsibility to opt out of this nonsense? Make it more expensive to send and maybe that will reduce the crap being sent.
They could just as easily make a rule that bulk mail costs twice as much as normal mail, and still require senders to presort them.
Sure, some senders would then try to send their bulk mail as normal mail. Then there could be another rule that defines what "bulk mail" is, and that anything falling under that definition must be sent bulk, with the consequence being fines and refusing to deliver that mail.
Yes, this is more complicated, and I'm sure some bad actors would still be able to get through, but I expect this would greatly reduce spam.
I imagine part of the issue with something like this is that the USPS is chronically underfunded, and policing and enforcing it would be to much extra work.
There's no reason why it has to be defined this way, though. Create a new category for "marketing mail" (or whatever), define what belongs to it, and require senders to continue to presort it, but make it cost 10x as much (or whatever).
This is never going to be perfect, of course, but anything that reduces the amount of crap I get in my mailbox every day isn't just great for my levels of annoyance, but also great for the environment. 95% of what shows up in my mailbox ends up in the recycling bin. Creating, printing, shipping, and recycling that material is a huge waste.
It's certainly possible to do that, but keep in mind that if the government (or an organ of the government) is in the business of deciding what category of speech something is, that is eventually going to cause a problem.
Just because there are legitimate reasons for bulk mail to exist doesn’t mean it should be discounted. Most bills are electronic nowadays, and for the smaller number of people that live in rural areas and maybe still rely on snail mailed bills, I’m sure another 30 cents isn’t going to break the bank, when the bank is making 20% interest on the credit card bill.
For the USPS, "bulk mail" is a term of art, more properly bulk presorted mail.
There are several categories: 500 pieces of first-class mail, 200 pieces or 50 lb as "USPS Marketing Mail", Parcel Select Ground (50-piece minimum per mailing), presorted and carrier route sourted bound matter (300 pieces), library mail (300 pieces), media mail (300 pieces).
The tradition of discounting certain mailings, especially publications (e.g., newspapers) and books dates to the very beginning of the US postal service and was established by Benjamin Franklin.
One example might be an energy company (they tend to be big) that will send bills to a significant percentage of any residential area in its catchment area, probably four times a year at around the same time for everyone. Now repeat for water companies, phone companies, local government et cetera.
Bulk, for a mail delivery company, refers exclusively to volume. You get discount on volume. Creating massive volumes of customized mails has been trivial for years.
> The question I have is, why does the USPS provide a discount for bulk mail at all?
Because those are their customers, and they treat their customers well. The hundreds of millions of USPS mail recipients, on the other hand, are the product. USPS works for their customers, not for the rest of us.
>‘You mentioned making the service better for our customers; but the American citizens aren’t our customers—about 400 junk mailers are our customers. Your service hurts our ability to serve those customers.”’
-- Patrick Donahoe, US Postmaster General
This is of course an evil fascist talking point though, because the USPS has a large union that consistently votes Democrat, and how could we do wonderful mail-in ballots without the wonderful US Postal Service? Don't be a fascist, mindlessly cheer for USPS.
And this is why privatizing essential government services (ok, the USPS is a weird public-private hybrid, taking on some of the worse attributes of both) is a bad idea. The USPS should exist to serve regular-Joe recipients of mail, not to serve the bulk mailers that pay the USPS to put literal garbage in our mailboxes every day.
But once they have to worry about revenue and income statements and all that, an organization is going to care more about the interests of the people who pay them the most money, rather than the people they're actually there to serve.
The privatization is the bad part? Never a piece of junk mail until...
Oh, wow. Look at this, never actually privatized.
> The USPS should exist to serve regular-Joe recipients of mail,
Can we get a federal telegraph agency too, just in case I need to receive some telegrams? You have no idea how horrible it is having to pay exorbitant sums of cash on private telegraph service.
The trouble is that the USPS has a union which is a reliable voting bloc for the Democrats. That makes it sacred. Even if it performs a service no one has given a shit about since the 1970s, everyone will gush on about how "what if I need to receive some mail" and pretend that mail service belongs somewhere other than a museum exhibit. I predict that every 10 years or so, it will be necessary to invent a new reason why we couldn't ever get rid of it.
> I see minimal unsolicited mail because it is expensive to deliver.
This is where the US gets it wrong: over here, it's much cheaper for a company to send bulk unsolicited mail than it is for a regular person to mail a letter.
> I do miss the old phone books - nice thin and absorbent paper
Oof, if it's anything like the paper -- and, critically, ink -- used in our old phone books, well... yeah.
Works very well in my experience. I never get unwanted ads. The only time this is mildly annoying is during election time. Parties usually distribute flyers/pamphlets with their ideas (which are otherwise known, but still interesting to compare to each other).
Speaking of which, wasn't there supposed to be a crackdown in the US? Something about the SHAKEN/STIR protocols? I'm still getting loads of spam calls, does anyone know where things got held up?
My understanding is that SHAKEN/STIR wasn't supposed to stop spam calls, but to stop caller ID spoofing. That way you could have "verified callers". Some calls I get on my phone (Android, T-Mobile) do have little checkmarks next to them, and I assume that means they had valid SHAKEN/STIR information.
Possibly at some point we'll be able to just blackhole calls that don't pass verification. But that won't really stop spam calls; it just means that they'll come from verified entities. I guess in that case you can go after them with complaints and get them fined in the cases where their calls violate the law, but that doesn't sound like a great solution... I have better things to do than chase down misbehaving companies.
Yes and no. If the caller has an existing relationship with the customer they can always call you. So your bank can call you. There is a register "Reservasjon mot telefonsalg og adressert reklame" that you can put you mobile number, etc., on and everyone who might call is obliged to update their own records from it monthly so that they avoid calling anyone on the register.
I very occasionally, perhaps once every year or two, get cold called in Norway but now they are always from offshore. A few years ago I got a couple of cold calls from Norwegian companies but when I pointed out that I had registered all the family's mobile numbers with Brønnøysund [1] they apologized and that was the last I heard from them.
You would think so, but the Republicans have been trying to kill USPS for decades, and have managed to pass laws which both mandate that it pay for itself without taxpayer dollars, and hamstring its ability to be profitable. Some of these hamstringing measures were temporarily removed in 2022 as an emergency measure, but the USPS remains without a guarantee of stable funding in the future.
The only thing more inefficient than a purely governmental organization is a private company providing services to the government. There seems to be a persistent belief amonst citizens that governmental services would make them more efficient, more responsive to customers, etc, but following the money suggests different motivations.
The reason it is inefficient is because it provides services that a private company would not, or would charge a ton of money for. The USPS has to deliver mail to everyone with a postal address -- they do a whole lot of 'last mile' deliveries for FedEx and UPS, etc. Getting rid of the USPS means that a lot of people would either not get mail or pay out the nose for it.
Some things just aren't profitable, and frankly shouldn't be. Let's not forget that.
Seal Team Six Bake Sale! GoFundMe for Black Hawk nav system retrofitting! Reservists doing Speedo/Bikini carwashes! I'm sure it would work out great...
Really shows they don't really care about infrastructure. The markets could collapse and FedEx, UPS, and DHL could go under, but because the USPS is still there, you can still send mail from one coast to another.
> but the Republicans have been trying to kill USPS for decades
A little disingenuous. Republicans attempted to force lower government spending by causing a deficit decades ago. The strategy (for those that were good willed) was that a large deficit would politically force spending cuts across the board, obviously including USPS.
No spending cuts have ever been made and the deficit has grown exponentially instead. Congress discovered debt doesn't matter, and now neither side cares.
Welcome to the uniparty. You can continue to pretend DC Democrats and Republicans are different, but they're playing you with fake controversies that never are resolved by Congress
> > but the Republicans have been trying to kill USPS for decades
> A little disingenuous.
No, it's 100% true. I'm not a Democrat by any means and I'm in fact highly critical of the Democrats. But if you look at the history of laws proposed/passed with regards to the USPS, there's absolutely no ambiguity that the Republicans are trying to kill the USPS. This has nothing to do with spending, either: the USPS has not been funded by taxpayer dollars since 2006, with the exception of the 2022 bailout. After 2006, the USPS operated profitably for a few years, but Republicans have passed regulations which prevent the USPS from operating profitably, leading for the need for the 2022 bailout.
This isn't about debt or spending, and it isn't a matter where both parties are the same. I'm not a Democrat, and I don't even like Democrats, but the USPS' funding issues are unambiguously and intentionally caused by Republicans.
I'm not sure what total junk-mail volume is, but the USPS moved 127.3 billion "units" (pieces of mail) in 2022, down from a peak of 213 billion in 2006.
First-Class mail volume was 48.9 billion in 2022, single-piece (which I presume means non-bulk) 12.9 billion.
Assuming an average weight of 100g per item, that's 6.7 million tonnes of marketing (junk) mail, give or take an order of magnitude.
A tractor-trailer rig can carry a maximum of about 45,000 lb (20 tonnes).
The junk-mail delivered in a year is roughly 400,000 such truckloads of mail. Or assuming 5-day/week delivery, about 2,000 trucks operating daily for a year.
(I'm using round numbers as the initial estimate is rough, just trying to give rough scope to the scale.)
Yes, I'm against federal subsidies in many industries. If it's a public messaging system such as USPS, why does it deliver so much spam? How about we just pay for it and stop that part? Everyone loves that with Fastmail vs Gmail - how about we do it for our national message passing?
> capitalism
How did you arrive at capitalism when I criticized federal funding? You think the idea of a federal govt and paying for stuff with public taxes is a core pillar of capitalism?
Chiming in with something I've posted in the past that I've found reduced about 80% of this spam mail and only takes 10 minutes to set up: https://news.ycombinator.com/item?id=31070730
For automatically dealing with the type of spam that is frequently seen in YouTube comments, I can't help but wonder if perhaps a continuously trained LLM would be well suited.
Youtube spam comments tend to have highly atypical patterns, using things like unicode characters to avoid triggering keyword and URL filters. Something along the lines of GPT should pick up on these things pretty easily, and could similarly pick up on the actions these comments are requesting of users "message me on telegram", etc. It could also probably detect when spammers are trying to impersonate youtubers.
It's not really the kind of thing that spammers can use LLMs themselves to work around, either. Any attempt to get past the anti-spam LLM is going to look quite unusual compared to the typical comment which would tip it off.
Strangely Google seems reticent to try something in this vein though…
I actually tested this at one point with GPT-3.5 just by finding spam and non spam comments on a series of Mr Beast videos and, yeah, it was pretty great at it. Even ones that I wasn't 100% sure about it echoed that but would lean one way or another. I asked for outputs like Confidence score|Spam/Not Spam|Explanation and never saw it mark a comment that I'd consider genuine as spam and vice versa.
Obviously this has a selection bias because I had to choose the inputs but there were some that said things like "I ate a ghost pepper on my channel" and stuff that were clearly spam but, to someone not aware that kind of thing is trying to bait you into looking at their channel, it'd appear as possibly just genuine. Heck it may have been typed by a human who owns the channel but is still spam. GPT got it.
I tested this after the video came out a while back from one of the larger channels pleading for Google/YouTube to do something about all the spam comments and the general consensus seemed to be there was "just nothing they could do". Testing this lead me to believe they just don't want to do anything because if it's simple enough that some rando in his house can craft a prompt and get some examples to test in an hour or 2 then a multi-billion dollar company should be able to do SOMETHING.
Would a service provider like YouTube have an incentive to shut down every single spam post? It's plausible that spam can drive up engagement in some cases.
I'd be curious to hear what gives you this impression. I've read that shadowbanned users on Reddit get banished to a sort of "heaven" echo chamber where they can see only bots replying to and agreeing with them, but I've not seen anything suggesting that happens for unbanned users.
There are loads of comment-copier/remixer karma farmer bots but reddit users are becoming better at recognizing and flagging those.
I can imagine a situation where someone creates a walled garden that people will willingly pay to be part of. Costs money, involves actual proof of identity (but you could still be broadly anonymous within the garden), with the value proposition being an elimination of all advertising. Web ads, spam, all of it.
There is probably some critical mass where this would work. Some people would pay to be able to just not have to do combat with the whole world simply to enjoy the Internet.
I would like to see this done using domain validated identities. It might not work for everything, but, if I can prove I own a domain, it's a globally unique handle that can be used to build online reputation and trust.
It would also make is possible for larger companies to attest to purchases made and / or the quality of participation within their community. Imagine a scenario where I donate $50 to an open source project using GitHub Sponsors with my domain as the identity and GitHub attests to me spending that $50.
Over time, it would be possible to demonstrate a significant "investment" in your domain validated identity and it would be done by spending money online like you normally would without any additional cost. The attestation that you spent the money is simply a side effect of something you're already doing, but it's a really good indicator (over time) that you're a normal participant.
At the very least, I think having a domain as a globally usable handle would help to reduce impersonation which is a serious, difficult to solve problem right now.
Well, the monetary system itself ticks most of those boxes, it's just very limited in what you can communicate. And it is not perfect either I suppose.
In fact there might be significant overlap between something that can support a monetary system and something that supports trustworthy communication. The best you could hope for by solely focussing on communication is that it doesn't collapse under the pressure of having to keep financial transactions secure. Except any secure form of communication is probably going to be used for transactions of some kind at some point.
The way Keybase handles identity was a good way forward, in my opinion, that could be extended upon but since they were bought up I’m unsure about its future and no one else is building off of it or copying it, that I know of.
I can imagine using levels of its trust model to handle the levels of email I receive e.g. this domain has X validations and my friends haven’t blocked it.
That almost exists with facebook except your idea is more strict with identity verification for all accounts and instead of anonymity facebook instead lets you use your real name or a pseudonym.
Figure out how to keep sites hosted off the platform available without advertising, say by a monetary agreement with either them or their ad platform, and I'll be the first in line.
Bots have also made using nearly any dating app a chore. All apps are ruined by it, what we need is some regulations and jail (or some equivalent) to end this.
There are middle grounds. I have a myriad of servers that accept email for any domain as fast as the spam bots can send it, meaning the bots will detect them as open relays. Some bots use tracking codes back to themselves to confirm the relay is indeed open but many do not. The SMTP prompt even says not to use it. Some spammers eventually catch on and start trying poorly to attack my nodes which leads me to believe most of the spammers are not very technical.
One that probably leads very quickly to WW3, as the protagonists will find out, that most of the scammers are in countries not on friendly terms with the west (and therefore they tolerate cybercrime against the west).
There are also plenty of scammer operating from the west, though, so that anti spam foundation would not run out of work even if they just limit their activities to the west.
The death throes of a doomed industry. Be it steel, horseshoes or advertisements, as profit margins drop production rates will increase to compensate. Then as the machine is running as fast an as efficiently as it ever has, suddenly the margin becomes zero and there isn't any more room to optimize. The entire industry suddenly stops overnight. I await that day.
That is a slightly different issue. This would be like counterfeit steel being sold as of it were the real thing. Or direct-to-consumer horseshoes that will never arrive.
Much of this spam is actively out to trick you, as opposed to legitimate players who have no margin left to give.
Thanks. Yet another thing I hate about my work machine: window edge spellcheck. It autocorrects without asking me. So I don't notice that I have misspelled something. It just gets corrected to some other word.
I notice that as operations like Twitter and Facebook attempt to get leaner and meaner the spam problem gets worse. I've never seen so many spambots on Twitter advertising porn and guns, and every third comment I get on Facebook gets replies from a supposed scantily clad woman who wants me to DM her.
I run a business Instagram account and my routine is: Make a post, add hashtags, refresh page, delete the spam comments. Its always similar accounts with similar messages and they don't seem to follow the post after it is initially shared.
They also have no real incentive to make it easier to combat, because that's part of their competitive advantage over self-hosting. As long as it works well enough that people realize the benefit they are providing (note that this does not mean that it's best for it to work perfectly, as that would be invisible), doing more risks worsening their position.
I think spammers are just excuse that Google hides behind. I had static IP address for a decade, never used it to send spam (I filered SMTP ports from NATed clients), made sure I'm not blacklisted on any RBLs, configured SPF and DKIM.
Yet suddenly my emails go to spam in gmail.
Now, there's no way for me to do anything, because I'm nobody.
As for spam, my bayesian filter works just fine, so if I can successfully filter spammers out, I think Google can as well.
I feel like Google is doing this purposefully to make people like me to simply give up, and observing responses, it is successful.
>so if I can successfully filter spammers out, I think Google can as well.
This is absurd. The problems Google face are orders of magnitude more complex than your individual mail server. Your server benefits from the fact that no one cares about it. No one will develop a custom spam technique to send you spam. They will for Google. There will be whole offices in India dedicated to working out exactly what slips through Gmail.
You benefit from a kind of herd immunity where no one even sends the kind of spam that would reach you because it's blocked by Gmail so not worth sending.
The spammers don't attack mail servers, but individual mailboxes.
Your personal gmail account has the same risk of receiving spam as mine, but Google has much more tools available to detect spam. For example they can easily see somebody is spamming, when large gmail accounts are receiving the same message. They can easily see which hosts are compromised proxies as they receive tons of mail, and seeing large number of e-mails sent from questionable hosts should be easy to spot.
They didn't really improve spam detection much for for a decade, because blocking small hobbyist mail servers is more likely increase their user base.
Those special farms you're mentioning, wouldn't work, if filtering would be individual per user, as it should be, because each person's mailbox is different and each person has different definition what spam is.
I use some publicly available spam IP blocklists and my server started rejecting mail from gmail.com because they ended up on one of the lists for sending spam. I thought it was funny.
When this happens, gmail informs the sender that the mail wasn’t delivered and they try a few more times before telling the user that no more attempts will be made.
> When this happens, gmail informs the sender that the mail wasn’t delivered and they try a few more times before telling the user that no more attempts will be made.
That depends entirely on how the receiving server rejects the mail. If it just drops it into the spam folder or /dev/null Gmail won't be able to know. If it responds with a non-transient error code then Gmail will (correctly) NOT try again and inform the user of that.
> I don't buy it. If a potential customer contacts a business first, and the business sends unsolicited messages to the customer, that reply shouldn't get dropped as spam. And yet it does.
You mean a scenario where the customer contacts the business over e-mail, and the business then starts spamming from the exact same address? Does this actually happen to a meaningful extent?
Google is Directly killing e-mail: my friends can no longer receive emails from our server even if they have me whitelisted, in their contacts and repeatedly marked as not spam.
Nor is any mail server that interacts with other mail servers.
Using an email delivery service to ensure mail gets delivered is probably the easiest fix for singularity2001's issue. Otherwise they're looking at changing their hosting, changing their IP, or fighting the nearly impossible battle with Google to get their mail to be accepted by its system.
>Users generally consider themselves better served if most unsolicited email they receive with no strong trust priors drops into a black hole.
If users really want that then they can just dump all email from addresses not in their address book. Some already do that. It turns out that most people actually want to be able to get email from entities they do not yet know.
Spam email would likely be reduced 99% if it cost the sender a penny to send each email.
To implement, establish an email header field for a token. The tokens are one-time use, and can be purchased from some token selling authority. The recipient can check with the selling authority if it is valid. If it is missing or invalid, it goes to the junk folder.
I had read some years ago that putting this "fee" on sending email wouldn't really deter the spammers, when they find one sucker (and they do find them, and there's usually more than one) they can extract so much money out of them that it would still be worthwhile to spend the pennies sending those emails.
Domain names and IP addresses aren't free. If we just immediately trash all email which doesn't have valid SPF + DKIM + reverse DNS, we place a small upfront payment requirement on sending email. Domains or IPs can be placed on a blocklist if they still send spam, thus making the domain/IP wasted.
The problem is that waaaay too many legit senders still don't know how to configure their email servers, so you end up with a boatload of false positives.
> The problem is that waaaay too many legit senders still don't know how to configure their email servers, so you end up with a boatload of false positives.
Ironically Gmail has started demanding DKIM or SPF, possibly slowly saving email from this mess of unidentifiable mail.
I don't know .. we do have ads on the Internet and these do cost money to run.
Can't you pay less than a penny a user to run ads in Gmail today ?
The idea is fair though I don't think it's far from where we landed with how the large email broker systems operate with the large email system providers.
Would it? Somehow it makes financial sense to mail spam out physically, which I assume costs about that much. Is the conversion rate that much better for physical spam?
Your post advocates a
( ) technical ( ) legislative (X) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(X) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
(X) Requires immediate total cooperation from everybody at once
(X) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(X) Asshats
(X) Jurisdictional problems
(X) Unpopularity of weird new taxes
(X) Public reluctance to accept weird new forms of money
(X) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(X) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
(X) Extreme profitability of spam
(X) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
(X) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
(X) Countermeasures should not involve wire fraud or credit card fraud
(X) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
(X) Sending email should be free
(X) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
(X) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
( ) Sorry dude, but I don't think it would work.
(X) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!
Gmail's spam filtering has lots of monopoly hardening convenient limitations. Like the fact that sending out to or not-spamming a DKIM authenticated non-gmail sender once isn't sufficient to prevent their future DKIM authenticated messages from going to spam.
Of course, spammers have significantly moved to using gmail and it just streams right through.
It used to have, but I would say it's been getting worse. The number of false positives is definetly going up and I would argue that that already gives us an indication of how important Google sees email. If they would consider email a channel that carries imporrtant information they would optimise to reduce false positives not minimise false negatives.
Gmail is subject to specific targeting by spammers in a way that fastmail is not. The returns for spending weeks or months finding a niche way through Gmail's filters are justified by the number of gmail addresses that can be targeted, which is probably 3 orders of magnitude larger than the total number of fastmail subscribers.
Are they the same target email address though? If not, surely that's not a fair test if the gmail account has been around for 10+ years or so?
(I have the same situation, an 18-year old gmail account and a 6 year old fastmail account, but the reason I don't get ANY spam at all in the fastmail account is I only use it for certain things and it's much newer, so I'd argue at least in my case, that's not a fair comparison).
I was on fastmail for about a year. It didn't filter nearly as much, and Fastmail as a service was constantly experiencing outages. Pretty much weekly. I would say it's an almost unanimously inferior service.
I did use it during that period (and still do). I've never seen an outage. Are they regional outages? Or do I just go to bed at the right time to consistently avoid them?
But the spam filter isn't as good as Google's, 100% agree with that.
Sounds like some super weird, extremely local issue. And yes, I’ve been using it as main mail account for (after checking) almost 9 years now, so that includes that period.
Not an entire year. It was 2-3 day outages, weekly, for quite a while. I've been trying to find the tweets, but as Elon took away tweet searching it's been a challenge to go that far back
edit I remember they were open about experiencing a denial of service attack over several weeks.
The same tech works in both directions. Spammer creates 1000 email variants using a LLM, spam filter collapses those 1000 variants back into easily classifiable embeddings
In the end it's content that matters, not the form that it's send to - I don't care if it's my grandma sending me offers for viagra pills, or a spammer, and I don't care about the language either - I just don't want to get such offers.
On the other hand, if there is an e-mail that may be interesting to me, I don't care if it was sent by a human, or by a machine - I want to see it in my inbox.
In other words - it's not about distinguishing human/machine written text. It's about distinguishing content that is worthwhile for me from the one that isn't.
Tighter integration between browsers, phones, and email could help with this quite a bit, I think. Default-allow every domain you give an email to, default-allow every specific address in your contact book (and maybe everyone you know on social networks?), default-deny everything else.
A decent first-pass solution to part of this might be to just have email allow every domain in my password manager.
I think the data's there to make this work a lot better, it's just that all the parts aren't talking to one another.
The thing is that humans might actually use Email to talk to humans that they have never met before and that should be a legitimate use case of email, but because spam has made it impossible to keep up with the volumes we are bombarded with that is no longer the case.
You're correct, but this is one of the reasons the megacorps dominate this space; they have the resources to do that integration. In contrast, what would it look like for an independent operator to roll out or maintain such an ecosystem?
People put up with millions of ads everywhere but a handful of spam mails a day and everyone looses their minds. Meh, I'd rather have the spam then be beholden to centralized gatekeepers.
Extrapolated from "Nearly 85% of all email is spam." At that volume, if your spam filter started with a random coin-toss you're more likely going to serve the user's interest than not.
Rejecting email takes more than zero bytes back over the wire. When the other party in the transaction is assumed hostile, the principle is to provide them as little information as possible.
The fact that they don't acknowledge it doesn't mean they don't drop it.
I've had to set up someone's mail server last year. All mails sent to gmail were silently dropped until we set up all the current buzzwords for the domain/email server. Then they magically started to show up.
Possibly we were lucky that "just" setting up SPF DKIM etc fixed it.
> I've had to set up someone's mail server last year. All mails sent to gmail were silently dropped until we set up all the current buzzwords for the domain/email server. (...) Possibly we were lucky that "just" setting up SPF DKIM etc fixed it.
SPF and DKIM are nowadays the very basic methods used to verify if your emails are spoofed or not. Not having them in place is as good as setting up a spam farm.
This used to be a cardinal sin for mailserver operators; mail should always either be delivered, or result in a non-delivery notification ("bounce"). Dropping mail on the floor was a no-no.
Bounces didn't become haram because of gmail but because of backscatter spam. You can still reject the mails at submission time but that changes where you have to run your spam filter.
> They do: the test emails were never rejected, but never arrived in the test gmail account either. Not in spam, not in all mail, not in the inbox.
And that's fine because that's exactly what SPF/DKIM were designed to do. The spam folder is not a dump of true positives. It's the bucket where you train the filter by evaluating somewhat likely false positives.
You have to understand that telling the spammers each and every detail is simply not viable. So you're gonna end up with situations where lazy/uninformed postmasters get caught in-between, sucks but unavoidable.
You have to understand a rejection is not telling every detail. And diligent postmasters have observed Google and Microsoft drop email from low volume senders with faultless configuration and reputation.
> Sometimes it's too much info to give out. You want to accept phish/malware and silently drop it so that the bad actors don't know you've detected it.
Users' interests are served by being informed when this happens. And plain text correspondence is not easily mistaken for malware.
> A perfect technical configuration doesn't make you inherently trustworthy or provide you with faultless reputation.
> The spam folder is not a dump of true positives.
That's exactly what it should be. If somebody sends me an email, it should arrive somewhere that I can see. If that means needing tiers of spam folders, each one spammier than the last, then so be it.
Over the last year, I ended a 20-plus year long run of hosting my own email (and email for a handful of other people and businesses I had a relationship with), entirely because of Gmail's behavior.
People here saying "it's spam, not Gmail" are being distracted from the numerous issues that independent mail services do have with Gmail.
Gmail is extremely uncooperative at accepting email from services that aren't Gmail, Comcast (sometimes), or Microsoft. You can have everything configured correctly, on an IP you've owned for years, and aggressively manage any outbound spam, and Gmail will still hate your guts and bounce your email or file it in the recipient's Junk folder.
Before Gmail got huge, email service providers typically offered an avenue for addressing false-positives in their filtering systems. Gmail really pioneered the "nah, screw you" approach to this.
Meanwhile, Gmail is itself a huge source of spam.
I (maybe perversely) loved hosting my email and email for a handful of other people. It's fun. Gmail took all the fun out of it and turned it into a seething hatred.
I just recently started self hosting email. I settled on relaying through gmail to all hosts except a whitelist of hosts I know won't blackhole me - mostly just a few niche domains of friends who do the same thing or small organizations they're a part of. Have you considered this? It's not optimal but it's a practical compromise. You get to keep your From: field as your domain too if you set it up right. I couldn't make it into Microsoft inboxes at all before I set up the relay but now I can.
- Make a gmail account just for your email server, which forwards anything incoming to your host in case someone emails it directly (you'll be able to discover this gmail address if you dig through delivered emails' headers but it won't be in the From: field)
- Let Gmail authenticate with your SMTP server in Gmail's advanced options, and make sure the options are checked to retain the From: headers of relayed emails
- Generate an App Password and set up your mail server software to use Google's SMTP relay with the email and app password
There are other services to do this too I'm sure, but I'm happy with Gmail for now. And you can always transparently switch it out to another service if Google pulls something.
I plan to write a blog post on the whole process of setting up a mail server to configuring it with a gmail relay like this.
I did consider that as an option, and I tried routing email through a number of other delivery services, but ultimately I rejected it for a few reasons:
I was hosting email not just for myself but for a few other people, and that gets tricky to route through Gmail;
Gmail could change their policies at any time and give me a really bad day, potentially when I can't respond to it in a timely manner;
If routing email in this way ever triggers Google's abuse mechanisms, then potentially I'm losing access to a lot of the Google network, and while I don't use it for anything personally, sometimes I have to work with companies that do;
After spending tons of hours dealing with Gmail-related headaches despite not using Gmail myself, relying on them to get mail routed felt like a deal with the devil.
I've instead helped a bunch of people get set up with Fastmail. I love Fastmail, they're great, I miss hosting my own email but they're the next best thing. Fastmail must be handling enough traffic that they're hovering above Gmail's piss-off threshold, and really my experience with them has been extraordinarily good. Everyone I've set up over there has been happy with them too, save for one person who got told by the next IT hat-wearer that "everyone's using Google Workspace and you should be too" (and then immediately ran into a problem during setup that snowballed into a big hairy mess).
Exactly this. I enjoyed the technical aspects of running my own mail server. I hated having to constantly manage black lists and dealing with "postmasters" to get my email to be consistently delivered. Some spammer manages to hack a server on the same IP block as you equals trouble for months. I just couldn't justify the time spent and the non-delivery any longer.
Even google is aggressive about IP bans on other services.
There was a time my home IP on Comcast was on some blacklist with good. Every video was run through captcha. Searches too.
Comcast was useless to give me an IP in a new block.
What solved it? I signed up for gsuite free at the time and moved my email (from a colo) from “on prem” to them. Suddenly my home IP that’s used to access their services is cherry and no longer suspicious.
Note: that doesn’t seem to work anymore though. I regularly proxy some traffic through a linode and google does the same thing. Everything behind a captcha that’s stupid difficult to clearly 60% of the time. I hate traffic lights.
I doggedly still run my 20+ year email platform for self-use. I ran into the same problems with Google/Gmail. I am ashamed to say that the solution I settled on was to just give up and pay them for smarthosting. I still host my own servers, but my exit pipe is through gmail. My life got easier.
The rule is pretty much every interaction you have in the Googleverse is easier if you just pay them something. It's pay to play internet, and yeah, it's a problem.
> It's pay to play internet, and yeah, it's a problem.
1. How much do they charge?
I'm genuinely curious. I don't self host, but use a 3rd party (fastmail). I send very few emails to people I don't know, so personally, I don't run into issues with having my email sent to spam.
2. I don't think paying in itself is the real problem. I think it's more a matter of who you pay and why you pay.
- You have to pay to register a domain name.
- You have to pay to host your own server (whether your using a hosting service or hosting from your basement)
- You have to pay to have gmail not mark your email as spam - ok, I'll admit, this is a little silly, but you also have to pay (via a stamp) to have USPS send letters to their recipient
3. Perhaps because so many people use and trust (whether they should or not is another question) gmail, it makes sense to pay in some scenarios? But obviously, for personal mail servers, I agree, asking to pay to play is a bit of a stretch.
> Gmail really pioneered the "nah, screw you" approach to this.
From my POV, Yahoo pioneered this way before Gmail. 10 years ago when I was managing outgoing mail infra for a company that probably sent 100k emails per day (mostly purchase receipts and subscription notifications), I was intimately aware of deliverability characteristics for Gmail and Yahoo, which were our two biggest destinations by a large margin (if I recall, Gmail was about 60% and Yahoo was about 30%). Gmail, despite being the overwhelming majority of our traffic, was hardly any problem at all, whereas Yahoo would regularly give us the dreaded 451 response that indicated that your mail was going to be held for 2-24 hours and then randomly rejected or accepted.
BTW, if I were to run my own mail infrastructure again, I'd definitely use someone like Sendgrid with a dedicated outgoing IP with a clean reputation. It makes a very big difference. We eventually ended up with our infrastructure split across multiple IPs by function (e.g. receipt emails were the only thing ever sent from one specific IP because we wanted to ensure deliverability).
The trick to get reliably mail delivery to Yahoo back in the day was to be spending several thousand dollars a month buying ads through Yahoo.
Then when your mails to Yahoo got repeatedly blocked by Yahoo and you got tired of having to repeatedly report the false positives and and get unblocked only to get blocked again a few weeks later you have the person in charge of your ad spending call up your ad sales rep at Yahoo.
You asked the ad sales rep "Why should we keep spending $X/month on Yahoo ads to try to acquire customers, when Yahoo mail keeps blocking the receipt emails, setup instruction emails, and response emails to support questions we try to send to those customers which angers those customers and they cancel (or even try chargebacks)???".
What happened then is the sales rep puts your ad manager on hold for a few moments, then comes back with someone high up in Yahoo IT management conferenced in, and explains the situation. The high IT manager puts you all on hold, and comes back in a few minutes with someone who actually deals with maintaining the spam filters conferenced in, explains the situation, and tells the maintainer to add your domain to a special whitelist of mail that is always accepted.
I suppose that the only real problem is sending, not receiving, nor setting up software.
If I were to set up my own mail server, that would send outbound email through Sendgrid, AWS SES, or maybe some other established and trusted sender.
Building trusts with behemoths like GMail or outlook.com is too involved an affair to be worth it in the general case, and not even through some malice on the side of the latter; it's just the reality of a medium not protected from spam.
I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one. FWIW, I used to get a fair bit of inbound spam from Sendgrid IPs, so they floated in and out of my greylist on the regular and I wasn't too surprised when they didn't turn out to be a good option for my outbounds. I was surprised about Mailgun; I don't recall receiving any bad traffic from them ever, and the couple of times I've used them, they seem to focus a lot of energy on not being a source of trouble.
> I tried both Sendgrid and Mailgun and couldn't reliably get mail to Gmail through either one.
I mean, you can't reliably send email to gmail from gmail. It's not really the origin that's the issue, it is that gmail spam classification is quite bad. Good emails from known recipients you've corresponded actively for years will continue going to spam in gmail even as you mark them not-spam for the thousandth time.
So yes, sending to anyone @gmail.com has decents odds of ending up in spam for no reason. That's just how gmail is.
But no, sending it from your own email server doesn't make things worse. I run all my own email infrastructure and sending to gmail addresses works just as well from my server as it does from gmail itself.
> Building trusts with behemoths like GMail or outlook.com is too involved
There's nothing like that actually. I have e-mail server running since 2000s. When gmail appeared, everything continued to work as expected and it did for many years. Around 2018 or 2019 I noticed that my e-mails started going to spam folder. I didn't send any bulk e-mails, I was not on any RBLs, I was not compromised (I actually was very careful to block outbound SMTP from any user except the one running MTA), and of course had SPF & DKIM set up.
It looks like their spam filtering is just arbitrary and feels like done on purpose to discourage running personal mail servers (and looking at comments it works really well).
Honestly, I think some kind of campaign is needed to put them in place, like starting blocking e-mails from gmail (maybe responding with a message encouraging to switch account). I remember in the past steps like that were done, but feels like today people are more acceptable of centralization.
They certainly are discouraging from running small mail servers, because it's harder to hold a small installation accountable. A spammer is usually a tiny installation which would disappear overnight, being run on random botnet nodes.
I don't think they explicitly are trying to quash smaller independent mail servers (personal or commercial), they just see what spam statistics show, and update blocking rules accordingly, maybe fully automatically, using ML. The fact that they also quash small-time independent competition is just a nice (for them) side effect.
They operate since 2007, they can track who is reliable and who is not. It's not like IPs change[1].
[1] yeah, they might be some people trying mail servers on dynamic IP, but I'm not talking about those and Google would have some justification not categorizing them as reliable. I'm talking about using business level ISP offering with a static IP address and matching RevDNS.
If google can't automate it then it doesn't exist for the most part. They bend over backwards not to deal with actual humans, and they're so big now they could not care less if they lose -1 human or business, it simply doesn't matter unless you are rich enough to know someone in the 1st tier executive team.
So, I've run simple Postfix + Django emails for transaction confirmations, password resets, and in a few cases, 2FA, for a bunch of different sites I've worked on since I started my freelance business 6 years ago. I've never had a single complaint that my emails weren't delivering, nor has an email ever gone to spam folder in testing. How did I do it? The answer is simple: I didn't send any spam!
The last time this came up on Hacker News, one of the top comments was something to the effect of "we did double confirmation and a variety of other measures to avoid being marked as spam, but ultimately the entire time we're just one bad email campaign away from being blacklisted".
One bad email campaign? Is there any other kind? That's just spam.
The user didn't say this, but I'd bet money their "double confirmation" starts with a default-checked checkbox with small text asking for permission to send emails.
Every time I've talked to someone who has problems with email deliver-ability, if I dig into what they're doing, it quickly becomes clear to me that they're sending spam, but they're so indoctrinated in corporate culture that they don't even know that what they're sending is spam. Here's some translations for you: Marketing email = spam. Lead generation = spam. In most cases, newsletter = spam. Sale announcement = spam. Promotion = spam.
I'm not claiming my experience is universal. I'm sure that there is a non-zero percentage of sites sending legitimate emails getting marked as spam. But it seems to me that more often than not, the reason your emails are marked as spam is that they are, in fact, spam. And most strategies people discuss for avoiding being marked as spam, are just avoiding the most obviously egregious forms of spam, and finding users with higher tolerance for spam.
I run a site for a large corporate client. People can sign up to get a quote from a regional dealer for a specific type of complex product. To get a quote, they have to fill out several forms, and select detailed specification, etc, and choose email as the way to deliver the quote.
We regularly have users flag the email they receive from this process as spam.
I have personally called to follow up in some cases to understand if our service was being abused or what the issue was. It was eye-opening. One user said "oh, yeah, I wanted that when I filled out the form but not when I got the email." Several marked the proposal as spam because they didn't like the final quote that was put together from their requirements.
Several said things like "I get too much email" and when pressed as to why they checked the box that said they wanted their quote delivered as an email replied that they didn't know, or they changed their minds, or they didn't want HTML email, or they didn't want a plain-text email, or their name was not in the subject line of the email, or that their company name was not in the subject line of the email.
This is a very low volume, very expensive, highly technical product. We're talking maybe a dozen requests per day nationwide. So those people flagging the emails as spam have a significant impact on the overall deliverability to services like GMail.
Yeah, unfortunately customers can be pretty shitty. We've had a lot of interesting experiences with this and also paid subscriptions - things like significant others accusing us of a massive fraud, soon afterwhich the person who signed up for the service to ask to be reinstated.
That said, if you can get people on the phone, they tend to be much kinder. I think most people think they're just shouting into some empty void.
I think trying to frame "customers mark unwanted email as spam" as a customer problem is just shouting at a river at this point.
How I design my, admittedly not corporate, email delivery system is to require the user explicitly click "Send this to my email" every single time. No email is sent without a user clicking a button painfully explicitly asking for it.
Honestly treating e-mail like sex where it requires enthusiastic continuous consent feels like it should have always been this way. It's really not that hard to have a button and have receipts/tracking to only be on your account page unless they ask for it.
Yeah but like, the guy who mentioned the customers marking emails as spam even though they requested it, did the exact same thing. They consciously solicited a quote and specifically asked for it to be delivered by email, and then for some unknowable reason, marked it as spam despite having completely 100% consented to receiving the email.
> for some unknowable reason, marked it as spam despite having completely 100% consented to receiving the email
Ordinary users aren't very interested in what is and isn't spam (nor are legislators, for that matter). It doesn't help that organisations like DMA lobby to tinker with definations. Unsurprisingly, a large proportion of users think spam is just "email I don't want". So they click on "mark as spam".
to be fair most people don't know that can get the email sender blacklisted or are simply ignorant of how email works at all other than checking it, answering it, deleting it, or sending it to spam. The sad fact is that a lot are afraid to "unsubscribe" because that is a common trick by fraudsters to get a hit just like allowing images.
i too run a quote service. only transactional emails, things users have requested. and still get flagged.
my favorite so far is when a user complained they weren't getting email and it's because they already blacklisted us so our mail provider diligently didn't send to them anymore.
I run an email domain for myself and three friends. I set it up 10+ years ago and originally it was for ~15 people. It's only personal email and no one uses it for marketing or anything even remotely shady. We setup DKIM and all of the similar best practices. In the last 6-12 months, we've had lots of issues with gmail marking our messages as spam for users we haven't emailed before.
And what sucks about it is there's nothing you can do. People assume it's your fault for being weird and not using Gmail or outlook.com. And there's zero way to contact Google or submit information to convince them you're legitimate.
I believe in an open, distributed internet. I don't think it's good that we're moving towards a world we're the core protocols that defined the internet are being replaced by proprietary versions controlled by a handful of trillion dollar companies.
This makes me sad the "old" internet has been dead for a while it seems. I also run my own email "server" but mostly only use it as an inbox so I don't know how things have evolved over time. At some point I signed up for some google feature that will send me a reliability report, zipped, xml from google. I don't remember the details and, yeah, I haven't ever tried to reach out to google but based on their products I imagine it is impossible to actually contact anyone.
You can be judged for your entire IP range if you use the wrong VPS. DigitalOcean is one such place. They put all their domains on spamhaus by default, which you can get removed, but it seems MicroSoft and some others still don't like you just for being from DigitalOcean. I never sent any spam from my IP that I had for a decade but ran into this when I tried delivering to MS emails even after getting removed from spamhaus. Setting up a gmail relay that retained my from: address did the trick though.
That's been my experience too. Google mostly accepts things as long as the domain isn't brand new, but Microsoft will just not work with you because they don't like the IP block you are on.
I’ve got a mail server running on DO. While Google sometimes spam bins Mail from my server, the server has never been listed on spamhaus. I don’t care about MS, so I don’t know how delivery there is.
The correct thing to do is user-initiated subscription signup. That means the e-mail subscription itself must be user-initiated and not part of or blocking some other flow.
"It goes without saying that our messages are not spam" haha no. I don't know what it is with these people who think their god gave them the inalienable right to send messages without rate limits just because they are signed with DKIM. The most likely explanation for why that site "School Interviews" got rate-limited is people marked their junk as spam and their sending IPs got bumped down into the bozo quota. And the most likely reason for people to have marked them as spam is they failed to do verified double-opt-in and just started spamming away at whatever address their customers mistakenly typed.
You're correct that a lot of senders have no idea when they're sending unwanted email, and that unwanted email is well within the realm of possibility here. But don't assume DOI is a panacea; you can use DOI and still send unwanted email. It can improve quality metrics (fewer bounces), but engagement metrics are a much stronger signal, especially for gmail.
Can you think of a scenario where a well-intentioned organization doesn't realize they're sending some unwanted mail, and by looking at the right metrics they realize they have a problem and take steps to fix it?
Same. I don’t wanna subscribe to any newsletter. If some website tricked me into accepting it, or just added me to the list anyway, it’s going to spam. Hope they enjoy the Gmail reputation penalty.
There's also the possibility that their emails are full of links, which is a decent indicator of a low quality spammy email.
There are exceptions, e.g., I sent an email full of research with sources to a family member I've been emailing with for fifteen years and it went to spam, despite it being @gmail.com to @gmail.com. In retrospect, I was misusing email versus sending a document or a link to one with it in.
Is it? If I'd sent a link to a document, they'd have the latest version and I could continue appending to it, and they'd have a version history and could even contribute to it if I permitted them.
The times have changed. It's not unreasonable to expect how people use email to have changed, ergo people sending emails full of URLs are statistically more often than not spammers.
Is it a broad stroke? Yup. But I'm willing to bet that I'm a fringe case and it prevents a ton of spam.
I deal with email at global scale, and yes, you're a fringe case. There are many billions of messages sent every day which have lots of links, and which recipients in general are interested in, and want delivered to the inbox (or promotions), rather than spam.
I agree. I also believe that message contents are much less important for abuse classification than nerds generally believe. Spam is about behavior more than it is about messages.
Speaking from quite a bit of experience, content can matter at Gmail if it's very obviously malicious/spammy, but not usually otherwise. Metrics matter more than content at gmail.
For other filters/ISPs, content can matter a bit more, but as a general rule for consumer mailbox providers, metrics are the primary thing that affects filter outcomes.
The real question is why obvious spam enablers like GetResponse and other email services don't get the same /dev/null treatment by Google. None of the email gatekeepers you now need to use to have your mails arrive do any verification (CSV import hello), yet they are obviously not met with the same bans.
Because those companies maintain enough "good" customers that it impacts real businesses if they get entirely shitcanned, and those businesses complain to Google from both sides.
This almost makes me want to go through the experiment of setting up a legit email server with all the various authentication mechanisms and see if I could even get one email through to google or protonmail
I just set up a brand new server with a never-before-used IP address. I set up SPF, DMARC, and DKIM, created a brand new account with a fake name, and sent an email to my Gmail account. Within 3 seconds of being in transit, the email landed straight in my inbox with 0% doubt of it being spam.
[Edit]: Forgot to mention that the IP is home IP, not cloud provider/hosting or something.
And as as sidee note:
the problem lies with the users and not with the email sender. For instance, if I send an important email to someone about an item they ordered from my site, and they tell me to send it to their gmail account, I connect to gmail to send the email, but if Gmail rejects it, I have fulfilled my part of the deal. If an email is marked as spam or rejected, it's on the recipient's end, not the sender's. The sender did not flag the message as spam before it is sent, the recipient did it after receiving it.
You absolutely can. Yahoo and O365/Outlook would be actually difficult, but you can build up reputation with those as well. Making it through small providers' filters will be your endless battle though.
Microsoft ban itself (azure) to spam, they use ml and while they cand add exceptions in general the systems are so complex and have so much hardening that is not posible our "messages are not spam "
I mentioned it before here on an old mail related thread: my Google powered inbox managed to flag as spam an email from Google Domains about an upcoming renewal.
Some things are just baffling. How can they manage to flag themselves as spam it’s beyond me.
If you were out on a walk and 9 out of 10 people where trying to mug you, you'd very quickly adjust your behavior to only walk in very safe places and let as few people as possible access that area.
There is a significant cost in spam protection by tracking reputation and content for the unending ocean of bullshit flooding the SMTP lines. Most providers want to cut communication with the spam source as quickly as possible to reduce costs.
Is spam also killing every social media platform, messaging app and even Google itself? Is spam not an indicator that people still use it if it's still lucrative spamming people, thus proving it isn't dying and is actually a sign it is still used?
Consider that email accounts are the go-to account recovery method for most services, and it's ubiquitous in biz. Also consider that you can prioritize specific domains or filter x domains to never go to spam, e.g., your own company's domain.
Any "death" is that people struggle with their own mailserver as a general rule of thumb. Does that thus mean email is dying? No. As the article says, perhaps independent email is, but it hasn't been in a good place for over a decade at this point.
Oh my god, yes. To all appearances they declared defeat in the Great Webspam War some time around '08 or '09 and their results have been markedly worse ever since.
In terms of end-user quality of result, yes, I agree, but they're still wildly profitable, ergo they are not dying. They're a business. Their pulse is measured in dollars, not quality or user sentiment.
I've been on Telegram since 2016 and I probably receive one spam message per 4 months. And I'm super active in a lot of groups. Possibly could be related to the types of groups you join though. I've noticed that 100% of the spam I receive on Telegram is about blockchain scams so perhaps being in tech or finance related groups (which I am not) could increase the risk.
Like with Snapchat, almost all of mine are camshow and crypto spam. It must definitely be some kind of demographic thing with what you're active in; lonely nerds for the camshow spam? Lmao. It also wouldn't surprise me if there's lists for sale of people of xyz demographics.
I kind of wished the 1 penny an email idea would have taken hold. As much email as I send $5 would last me years. It would have probably deterred the people sending out millions of spam mails.
Would Spam stop if people stopped responding to it? There has to be a non zero amount of stupid people that react to junk mail and make a purchase or fall for some scam. This number is only increasing with more people coming online.
>Would Spam stop if people stopped responding to it?
I don't ever intend to respond to spam, and have become extremely adept at spotting the patterns and swatting it away. However, it becomes a game of chance, when a service like Outlook puts it right at the top of the app (both iOS and Android) where you would reflexively jab at it, unless of course, you pay the premium to remove it.
For now, I have found a way to stop this nuisance. However, MS are playing fast and loose with their policies and now very legitimate looking spam is leaking into the inbox, escaping any filters. Since last year it is appearing along with the glaringly obvious Unicode riddled ones, with increasing regularity. It seems like a matter of time and co-incidence, where you would end up interacting with a piece of disguised mail you were expecting e.g. an order from Amazon or a service which you use regularly, and possibly respond without checking the header.
This recent episode was probably the worst experience, albeit not the first time it has happened.
There is a wide spectrum of unsolicited mail, not all of it is stupid people responding to scams. I suspect the quality and response curves are inversely related.
From my experience (admittedly with independent email services that were around before Gmail was even a gleam in Larry Page's eye), Gmail is only a modest fraction of the problem. Other big players - especially Microsoft - are generally worse.
Flip-side, there seem to be more spammy messages sent from @gmail.com addresses than from any of the other email A-listers.
Agreed. I self-host for my and my friends' personal and project domains, and delivery to gmail works. Granted, nothing is commercial and the volume is so low that rate limiting is not an issue, but if you set things up properly, they'll take your mail, and if you don't, they're pretty good about telling you what's wrong.
On the other hand, it's simply impossible to satisfy Microsoft. We're irrevocably tainted by being in a netblock of a well-known provider, despite having held the same IPv4 address clean for over a decade.
For what it's worth, I managed to get whitelisted by Microsoft a few months ago after... 15 years of undeliverability or so.
I followed the process, and then kept insisting a bit by answering the emails saying they were not going to do anything and I had to check if I was complying with their rules etc. After two emails I had a real person answer me, and a few more emails later (basically insisting I was already enrolled in their various bullshit spam reduction programs and there was zero spam problem with my domain) I got told that I had been whitelisted.
As a user, O365's default spam filtering was just terrible about 2 years ago. I got so many false positives that I had to check my spam folder multiple times per day. I ended up adding very aggressive domain whitelists because I was so tired of it.
FWIW, a client of ours got their office mail server off MS's blacklist a few years back. In less than a week. But that seemed to require their ISP (a mid-sized firm in the Midwest, with awesome customer service) going to bat for them with MS.
I'm guessing some of the issues are just from Google being random about what it considers spam no matter who it comes from. I remember a comment on a somewhat recent thread from someone who had to move their business mail away from Gmail because Gmail would classify mail from one paid account at their organization to another under the same organization as spam.
I was just going to say that for the past year I (my server for the whole family) sent maybe 10 messages to google and I personally received at least 15 spam messages from gmail with SEO offers and other kind of spam.
Maybe some strategies from other industries could work here.
Such as a “bonded trust” system.
So a new email provider could use real money as a proxy for trustworthiness, since they obviously don’t have a solid history to rely on. For example, the major providers could demand depositing $1 USD per email/per day they want to send out in exchange for the spam filtering to be turned off for their domain.
That is if they wish to send out 1k emails/day to Gmail addresses and make sure they land in the inbox, they deposit $1k USD with Google.
The catch being that if more then 5% of the emails (or whatever the ideal percentage is) are marked spam, then their bonded money is taken away. And they’ll have to put up a new bond.
That way new entrants can get a foothold without having to jump through so many hoops.
99% of the emails i've received from companies that aren't strictly transactional are not ones that i opted in for, and no, burying it in the terms and conditions isn't "signing up for it". you can hardly blame the users here
That's actually pretty much original definition of spam email: the narrower definitions were an attempt by industry and government to operationalize subsets of the problem that could be regulated or systematically addressed.
not even counting all the instances where transactional notifications are marked as spam by heuristic, the message silently rotting away in a junk folder and contributing to a bad rep of the sender, simply because no one bothered to un-spam-mark it.
Then smaller email providers will have to be very picky about who they onboard.
And/or the senders of such emails will have to be careful that every email is legitimately and actively desired by the recipients.
Or there is no viable business here and we go without the extra emails, a not too bad tradeoff.
Sorry what year is this? How is this a question still worth asking? So many people already asked this year after year after year and the answer keeps being "no, ISP and the law did that when we locked down port 25 because you don't remember the INSANE AMOUNTS OF SPAM we used to get". Or of course, never lived through that.
Things like hotmail, and then yahoo mail, and then gmail won out because crime decided to fuck it up for everyone else. Thanks crime, you sure did your thing.
The other thing that bugs me about these articles is the brazen two-faced argument where email is at once an open, distributed protocol between independent peer operators, and also there's exactly one way to do it, the way German privacy zealots insists on doing everything, and there's not another way!
If Google wants to receive your traffic at a later date thats their business and not yours. It's an open system where sites set their own policies. Access to a site's eyeballs is not the right of an outside sender!
Spam (and the resultant filtering) is killing independent email, and it's an ongoing problem on Big Company Hosted Email too.
I barely use email anymore. Everything at work is mediated by Slack or Atlassian. With friends and family it's almost all text messaging. My kids' schools and sports teams use a bunch of different proprietary web and mobile apps to communicate with parents.
If Gmail and other big providers reject email with correct DMARC, DKIM and SPF then these technologies are not doing their jobs. Why have this if you can't trust them? We need something new here, something really secure.
I am self hosting my email and had the luck that after setting up DKIM I'm no longer being sent to spam. I think it worths the effort.
I'm running self-hosted email, and Gmail users have no problems receiving it (spf, dkim, etc. are all working, I guess I'm also lucky to have used the same IP for a very long time). But what is funny, is that most of the spam I receive, and that isn't cought by spamd, is actually from Gmail spam accounts.
One problem is that with GMail taking over so much of the world the spammers have become highly focused on defeating their filters. Worse, they seem to be slowly but continually increasing their success rate, all while non-spam is ever more frequently choked out by false positives.
The end condition of this race is only spammers will be able to send mail to Google, no legitimate users will have the time or budget to figure out how to get past all of the blocks.
There's a problem that's just as bad, it's the reverse POV of this article.
Basically, you can't block the big providers - Gmail, Microsoft, AWS SES, Mailchimp, Mailgun and friends - because everyone and their dog is using them. But their reaction to abuse reports is spotty at best... you're stuck between a rock and a hard place.
The root cause obviously is spammers and scammers, but governments don't care about putting a final stop to bad actors.
>> you can't block the big providers - Gmail, Microsoft, AWS SES, Mailchimp, Mailgun
Why? Respectable businesses send from their own domains. Friends and family never send emails nowadays, there are messengers for that. Anything from google goes straight to Junk folder.
Gmail might have a monopoly over consumer email but Microsoft has one over enterprise email.
B2C email is quite competitive with dozens of services.
Overall I’d say the email ecosystem is relatively healthy. It’s more competitive and interoperable that instant messaging with greater security than SMS.
So many comments here saying that spam is killing e-mail, but even with very light spam filtering on a non-gmail account maybe 1 spam message a day gets through? I get more physical junk-mail than I do spam.
Running SA, I get maybe half that per day and false-positives are extremely rare. Get about 1 false negative per day as well (which is fine; I could live with 5-6 really).
It probably depends on how long you've had your email address. I've had mine for 20+ years would be borderline unusable without spam filtering.
It's been much better since I took the time to set things up so marking and email spam automatically fed it into sa-learn. I still have to have a handful of rules to filter out senders who are "legit" enough to make it through, but ignore unsubscribe requests.
I'm about to venture into setting up a new email server. Mainly in that I'd been wanting to play with WildDuck.. and second in that I'd like to stop paying to relay though SendGrid, which I've been doing the past several years.
I setup a dedicated server not on a major cloud host, and am not looking forward to all the details involved in the lack of trust starting out. Let alone the dark art of spam detection. But I want to get back into it if only because I don't like how the major parties are cornering things up. I also want to be able to actually handle mail for several domains and not have it nickel and dime me to death. It costs way more for a single email account these days than it does to run a few dozen minor websites.
While it's nice that Google Domains (when you use their DNS) and Cloudflare both have included email forwarding, sometimes you want an actual box to send from too. And with the partitioning that GMail now does, I can't find anything anymore without hunting for it... the only benefit is two of the subtabs, I'm able to just delete all once in a while.
I wish that email were much more reliable and able to actually setup 2-way relationships similar to IM clients. And of course, limit/remove third party info sales/spam in those relationships.
If you're only a small time sender lack of trust is permanent. I've been self hosting for three years now, never sent spam, never had the server breached and GMail and Microsoft both still send mail to spam.
At least Microsoft no longer outright 550 refuse my mail so I have that going for me which is nice.
It's not permanent. Polish up your sending practices, slowly build up sending and you can deliver to all big providers from DO or OVH for all you care. It's the small providers that'll give you the most trouble.
Last time I tried, Google and even O365 delivered fine... Hotmail and Outlook.com, ironically considering they're also MS managed were the biggest problems from my testing. At this point, I'm willing to let some of it go to spam, as I think most people wind up having to check there regularly anyway. There's a lot of false positives, but so much actual spam and uce I don't really blame the carriers.
DKIM, SPF, etc only go so far anymore, and even graylisting doesn't seem nearly as effective as it once was. Again, not really looking forward to parts of this, but I do want to at least try self-hosting as much as I can. I like the cloud offerings from MS/Google, but don't like the companies or their actions in and of themselves. Only one real way to push back, and that's to actually try.
And on the small providers, yeah I can get that. I'm a bit more than a decade removed from self-hosting much of anything, and even then balancing DNS RBL with other factors was at least interesting if not frustrating.
Everything else is a “hack”. SPF is irredeemably broken (no possibility of forwarding email - ARC doesn’t work except for (again) huge providers like Cloudflare.
I send a DKIM-signed email? You KNOW it’s legitimate, no matter how it arrives…
Edit: I’ve been running MTAs for decades; first one in 1987 (yes, the UUCP days).
Many comments here say spam is the culprit. But spam has been a solved problem for two decades.* Ironically, Gmail doesn't even implement the solution: private, individually-trainable stastitical filters.
* In fact, I'm still using the filter I installed on my machine in 2003.
Most of the spam I get is via Gmail.
I just looked at my spam folder, and everything there is either from gmail or has such blatant spam properties ("Content analysis details: 62.6 points, 5.0 required") that it was easily routed to the dump.
Google is so bad that it's worth soft-blocking anything from gmail.com Reply to any new gmail address with an autoreply that sends the sender to a web page for authentication. If they don't jump through the hoops to talk to you, discard.
A heavy portion of the spam I receive are in my personal domain email hosted on mailbox.org from @gmail.com addresses.
Most of the email to my old @gmail.com that I check once in a while are from @gmail.com and a lot of them are not even in spam folder.
I get spam from new startups (in India it’s kosher to spam) mostly in “hyper whatever segment” that I know for a fact are hosted on Google Workspaces and still no amount of marking spam or reporting seems to work.
So at least in my personal experience it’s almost entirely Google screwing email things up.
I host my own email server (postfix, and dovecot) and I have DKIM/SPF setup correctly, and have helped close to a dozen others I know setup theirs.
I have had ZERO issues.
What I believe, but not certain, the issue others may have revolves around the host record, and reverse lookup of an IP address of your mail server. If the reverse lookup points to something other than your CNAME record, Google doesn't like it, and it gets flagged.
For instance, my mail server has a CNAME of host.foo.com, and mail.foo.com, and I have two aliases imap.foo.com, and smtp.foo.com. I have had my service provider change the reverse lookup for my mail address to host.foo.com, where it was xxx-xxx-xxx-xxx.location-att.swbell.com (or something to that affect), as multiple services are on that IP address. I only have a block of 5. The smtp server needs to respond with the reverse lookup name. So for postfix, the smtpd_banner needs to be host.foo.com. Google does do a reverse query to validate the user, and domain, most of the time.
Also, Google maintains their own internal DNSBL. If your email is getting flagged, and the above is correct, then the IP address you've obtained from your ISP has been problematic in the past. Email Gmail to ask to be removed with the new domain, and don't do it until the reverse lookup and pointer records have changed.
You also need to contact the various spam black list sights and get your IP address removed. If you SPAM, you will get blacklisted. I also host a blacklist.
The author of this article should have known about this, and wrote a very misleading article.
To be fair, most of the actual useful stuff that I receive to my gmail accounts are sent to spam. The only stuff that routinely gets through are the newsletters and stuff I've managed to get signed up for one way or another over time.
A friend emails me for the first time in a while? From a gmail account? Spam.
An receipt from my ISP? Spam.
(these are actual examples)
But I reliably get every stupid newsletter that I've ever signed up for even though after 12 years I've only opened 1 of them.
My wife and I were in Prague and she needed access to an email. When trying to login to her account provided by our home internet provider back in the states, she was denied access several times. So she called the ISP, after several lengthy calls they couldn't connect her to her account! From now on, both of us have moved anything important to gmail because we like to travel.
Spam is killing independent email.
gmail is fairly good at blocking it, is easy enough to get into, so it's been winning for years.
I used to run independent email. It took constant work to get close to gmail's level of spam blocking. So I switched. Found most alternatives weren't anywhere near good enough, and I don't have enough hours in the day even for my own email.
> Spam is killing independent email. gmail is fairly good at blocking it, is easy enough to get into, so it's been winning for years.
> I used to run independent email. It took constant work to get close to gmail's level of spam blocking.
This is very contrary to my experience on both fronts.
gmail spam filtering is mediocre. Spam gets through and good emails are flagged as spam. It's not very good.
For my own domain where I self-host my email, my spam filtering is quite a bit better. Approximately nothing is ever mis-flagged as spam (has not happened in years) and the spam that gets through is a tiny fraction of what my gmail address gets. So for me my own spam filtering is much better than gmail.
And no, it doesn't take constant work. In fact doesn't take any work at all. I set it up years ago, the bayesian filtering goes through all the mail but it takes zero effort from me.
I wish this myth that gmail has some incredible filtering technology that nobody can replicate would just die already. Gmail isn't terrible at spam/ham filtering, but it's not that good either, just mediocre. You can do much better with minimal resources on your own.
This has been getting consistently worse over the last few years, in my experience. At least on the user end, i.e. far more false negatives getting through.
True in my experience. Doubly so when dealing with foreign languages. I run a pretty international business with a lot of different email distributions. It's actually the spam filter in Google Groups that seems to constantly get tripped up with non-English, and in particular non-Latin, characters in emails. It puts far more of them into spam than it should.
I noticed a correlation between the volume of spam I'm receiving to my spam folder, and the increase in false negative arriving in my inbox. Spam volume is increasing in my experience.
I think it's been true for a long time that most of the spam doesn't hit your spam folder either, so this is not the signal you suggest. It could mean more false positives at a lower layer though.
I also monitor email at a network infrastructural level via CloudFlare for the domains, and the volume has definitely increased overall, quite significantly, including on domains belonging to clients. But yes, the amount that actually ends up in the spam folder itself is far lower than what actually gets sent to us.
There's definitely an uptick in forgeries, which is also probably why Google has started strictly enforcing the existence of SPF or DKIM even without DMARC.
plus it doesn't seem to tune. no matter how many times i mark steam or fedex emails as not-spam, i just have to check there every now and again these days.
Years ago I owned a one bedroom condo, had a baby, and moved to a larger rented space.
We were ~10 emails deep with a prospective tenant in our old place. I was using fastmail through a private domain (me@mywebsite.ca), the tenant was using gmail.
Everything was finalized, and I gave some particulars about how the tenant could pay us. This was enough to be dropped into their spam folder and, except for a chance encounter at the grocery store, the tenant would have believed we had ghosted and missed out on the condo.
The silent drop after established back-and-forth is frustrating enough, but the worst part was that we never managed to reestablish email communication. Marking my emails as not-spam, adding me as a contact, etc etc. No future mail from my address to his ever landed in his in-box.
I consider myself lucky that my email address was only locally blocked - gmail users in general can still receive email from me.
There are a million wrong ways to self-host email, which are usually default, and there's a lack of standardization between all the different servers and clients. Like, it shouldn't even be possible to accidentally send unsigned email, and every server should totally reject it (Gmail still accepts it!). Any old or insecure modes should've all been banned, kinda like how web browsers force min versions of TLS and heavily discourage plain HTTP. I don't have a lot of experience self-hosting email because of how quickly I lost patience with it.
Had more experience with XMPP, which had similar problems. If you're going to make a federated protocol, it has to be strict, otherwise a big player will lay down the law instead.
Yes. I no longer use fastmail for important emails because it cost me an awesome job once when it went to the employer's (gmail based) spam folder after being delivered fine for a week. It's super annoying to switch back and forth but gmail also lacks a critical feature i dont want to give up
Out of curiosity, are you sending from a node you own and manage, or do you have your email hosted via a third-party (either with your branding skin on it or with the third-party's branding skin, i.e. are you "me@corp.com" or "me-corp@gmail.com")?
So my employer, for instance, "has our own email" but it's just Gmail and we never have problems sending or receiving because we're piggybacking on Gmail's "This is a corporate account with several years of good behavior under its belt" trust signal.
My friend I have sad news for you. Email died long ago of old age but most people loved it so much they re-animated it's rotting corps and make it work like it's 1989.
Modern messaging revolves around the ego if it's creators, as a result it is very difficult to have it become an actual standard that others on the internet agree with and that is independent of their infrastructure and CI/CD mindset/process. As a result all possible replacements are only usable by tribes of people who fancy a client.
Naive hostility against censorship and corporate/government middle-ware also means these modern protocols are consumer grade or only usable by entities that agree with (in principle) the political convictions of the protocol creators.
(running my own email server since 12+ years) The most effective spam "filter" I use is a simple DNS check ;) No reverse lookup to the hostname the sender pretends to be? No connection. Simple. That simple rule eliminates almost all spam. SPF/DKIM checks take care of the rest. For low-traffic email servers the dns lookup costs are negligible. Sometimes smaller IS better :)
Email is outdated. It was not designed for the hostile environment the Internet is today. It doesn't even do authentication or encryption without extra layers of grease that nobody uses or supports.
There's the Dark Mail Alliance[0] effort, but almost nobody talks about it, while it should be a priority to get a new email standard finished and deployed.
Thankfully none of these efforts ever really go places, because they'd create a massive amount of churn for minimal net gain. Spam is primarily a social problem not a technical one
How many neew communication standards have there been in the last decade that haven't been horribly centralized? Keeping email alive is our only hope for open communication.
gmail did push me to become self-hosted (currently a PI, but should become a risc-v mini-computer in the near future) because they broke noscript/basic (x)html browser support for re-authenticating my gmail account. It was years ago.
Ofc, I did write my own smtp software (excrutiating simple), for the moment simple C (c89 with benign additions from c99/c11), but the target is 100% risc-v assembly with excrutiating simplicity.
For now, I may not be one of their enslaved user anymore, but in interop with self-hosted smtp server, gmail is probably one the less worse out there:
- they don't block (many smtp admin abuse of the broken and toxic spam-haus list by blocking instead of using grey-listing).
- they have IPv6.
- A few years ago it was fine to send them email with a pure IP smtp (yep, smtp was made to be able to work without the DNS mafia).
But there is a catch: you end-up in their spam folder, DNS SPF and legit email exchanges doing nothing about it (DKIM is excessive overkill). Here, the real issue is non-tech-savvy people must be aware that legit emails WILL end-up in their spam folder: gmail should have such a warning right next to "spam name" like "spam(must be checked for legitimate emails)".
For some reason (about 6 months ago) I started receiving about 100-150 spam emails a day on my iCloud account. Nearly all gets correctly classified and moved to the spam folder. But now I have to scan through 100s of emails to check for false positives… this does take a non-trivial amount of time, especially when you’re too busy to do this for a week or so… it builds up quickly. Anyone have a good strategy to cope with this problem?
> Will I do the due diligence of receiving and looking at the SPF and DMARC reports you can get about your email? If not, stop. These are daily (or weekly) emails from other domains about any issues they saw. You need to pay attention to these and if you don't, you do so at your own peril.
Do we really have to pay attention to these? I have an email account set up just to receive these. 50,000 unread dmarc summaries later... all useless spam that says all the messages passed.
Many of the responses here saying that it's spam, not Gmail, that's killing independent email are implicitly saying that Google has no way of differentiating between a server not being well known and messages having spam content.
This is saying that the problem is unsolvable. This is patently untrue.
My email servers are configured to do all filtering on whether the connecting server is properly set up:
1) Does the HELO / EHLO name resolve in DNS to the address of the connecting server? If the answer is no, then reject as spam.
2) Is the connecting server's IP on any of a number of more conservative anti-spam DNS-based blocklists, like those that are based on dynamic IP pools, or on spam honeypots? If so, reject as spam.
3) Does the SPF for the sender's domain fail? If so, reject as spam.
The amount of spam this eliminates is tremendous, and most spam that still gets delivered comes from the big spammers: Google, Microsoft, Amazon.
I do not filter content because I'm adamantly anti-spam and and talk about and share spam with other anti-spam advocates, so content filters would be stupid.
Speaking of stupid content filtering, the number of abuse addresses which have anti-spam content filters is ridiculously high. Companies should be embarrassed that they don't know how to run email servers properly and can't accept abuse complaints properly at their abuse addresses.
Google is one of these.
Also, Google doesn't appear to do the tiniest thing with abuse complaints sent to them.
Finally, Google doesn't give people information about their spam filtering, nor ways to adjust it, so as long as Google applies arbitrary both to server reputation and to content filtering, with no ability to adjust, self hosting and smaller email servers will suffer.
Google knows this, and they COULD change this, but there's no profit, no business motivation to do the right thing. They have an interest in NOT doing the right thing, so we can't expect them to care.
What we can do is we can remind people who use Google for email that their email is non-deterministic. Nobody can say for sure whether email will be delivered or received consistently, because no regular humans know Google's rules for filtering, nor do we have access to Google's email logs.
When there are problems, we have to remind Google email users that the problems are with their choice of email hosting, and that's the price of giving up freedoms for "free" email.
I'm always fascinated about how, no matter the community, we can see downvoting without any responses that show where someone's posting is supposed to be incorrect.
I'd be curious to know if someone disagrees, and particularly what part anyone things is wrong. I have years of evidence, but the evidence is from running my own server. I'd love to hear perspectives from people who have different sources of evidence.
Downvotes without saying why just seem... emotional. I admit I downvote people who make generalizations without backing them up, but I'm not sure what's happening here.
> This is saying that the problem is unsolvable. This is patently untrue.
That is a very bold (and false) generalization. Spam is not primarily a technical problem, it's a human one. Human problems have mitigations, workarounds, not solutions.
> 1) Does the HELO / EHLO name resolve in DNS to the address of the connecting server? If the answer is no, then reject as spam.
Again, too bold.
> 3) Does the SPF for the sender's domain fail? If so, reject as spam.
Bold and incorrect.
> I do not filter content because I'm adamantly anti-spam and and talk about and share spam with other anti-spam advocates, so content filters would be stupid.
Only works on a very small scale.
> Finally, Google doesn't give people information about their spam filtering, nor ways to adjust it, so as long as Google applies arbitrary both to server reputation and to content filtering, with no ability to adjust, self hosting and smaller email servers will suffer.
Any provider with any significant size won't give you the full details. It'd be the spammers' dream.
> That is a very bold (and false) generalization. Spam is not primarily a technical problem
If you read what I wrote, you'll see I was talking about this problem: "that Google has no way of differentiating between a server not being well known and messages having spam content"
What you call too bold and incorrect I have years of data showing otherwise. From where do you get your data showing that HELO / EHLO checking and SPF failure rejection are detrimental?
What part of not using content filtering works only on a very small scale? Very small scale of what? What does the scale have to do with the kind of filtering?
So you believe that if providers shared their criteria, it'd be "the spammers' dream"... Except that hasn't happened. Many providers explicitly state their criteria, because they inform their customers what kind of spam protections are in place. Only large providers play games with random, unseen and unknowable rules.
But if you have evidence about how spammers have been in dreamland when learning providers' criteria, please do share.
> What you call too bold and incorrect I have years of data showing otherwise.
It violates standards, unless the letter doesn't pass DMARC checks an SPF fail is not sufficient. You don't have the volume necessary for proving anything either, I'm sorry. The fact that you think content filtering is not necessary proves it.
> What does the scale have to do with the kind of filtering?
It changes how accurate your tests have to be in order not to flood your customer support with complaints.
> Many providers explicitly state their criteria, because they inform their customers what kind of spam protections are in place. Only large providers play games with random, unseen and unknowable rules.
You're just excluding all providers with significant user base, of course then the problem is simple and processes (if there even are any) transparent.
The moment you have any significant userbase you're going to have a bunch of people trying to bypass your filters. Be it for spam or phishing. No reasonable provider wants to reveal all those details, don't be naive.
Handwaving, along with no data. Also, you have zero idea about my volume or the volume of my clients.
Also, you assert, again without data, that knowing a provider's criteria for acceptable email is what would justify the energy needed to try to bypass it. Imagine that!
"Spammers: you have to not use spammy servers."
Spammers: "Let's start using 100% completely legitimate email servers."
Sorry. I think you just don't understand how ridiculous that is.
> Handwaving, along with no data. Also, you have zero idea about my volume or the volume of my clients.
No, not really. I'm basing what I'm saying on best practices outlined in many places, standards and well yes, also my experience as mailop.
You're the one going against the entire industry with things like content filtering being not necessary... I won't start explaining again why that is simply wrong or just a sign of the small scale you operate at.
The problem, as I see it, is: do I have the _right_ to send a legitimate, nonspam, email? And do providers such as google have a _duty_ to deliver legitimate email?
Yes, providers do have a duty, unless they inform their customers that they intend to do otherwise.
Google thinks they're too big to either be a responsible Internet presence or to inform their users that they're going to do whatever the hell they want, and the rest of the world can go screw itself.
It'd be nice if there was a directory of well known good email servers that you need to pay to register. The payment aspect should be enough to keep spammers away. Then, companies like Google would have less of a justification to implement these practices that end up hurting small players.
I switched off gmail to a paid email provider exactly because of incoming delivery problems. One too many important messages had been aggressively filtered, and I was about to apply for a job. No way I was going to leave those emails in the hands of a filter I didn't trust.
Running my own mailserver with all the correct standards and clean but low traffic IP has been always been find with GMail. I have just given up with Hotmail et al. The is nothing you can do where they will accept mail for more then a week.
Isn't this the original problem Bitcoin (hash cash?) was supposed to fix? If you send email, you do some sha256 computation to make the cost of sending an email significant, but not cheap. Spammers would go out of business.
I think there’s a related problem here: spammers (often doing cold outreach) are very happy to use gmail to send their wares. Gmail provides no
mechanism to independent mail servers to report those people.
The true problem, I think, is: do I have the _right_ to send a legitimate (nonspam) email? And does an email provider have a _duty_ to deliver legitimate emails?
Reference? I googled but couldn't find any data, other than a very old report claiming 47% usage. (Not the same as "properly" but who knows what you really mean by that.)
Also, I wouldn't be surprised if less than two percent of all domains are non-junk domains anyway.
> Reference? I googled but couldn't find any data, other than a very old report claiming 47% usage. (Not the same as "properly" but who knows what you really mean by that.)
I very much dispute reject/quarantine being the "proper" configuration. Yes, it is the strongest configuration. I had to back down from quarantine in the past due to the way some customers wanted to forward our mail (breaking SPF of course but also breaking DKIM, thus not forwardable with DMARC p=q). Even without p=q there is still a signal and MUA (or with server assistance) can flag the mail as suss, so p=none is not "wrong" or even "inadequate". It's just ... less than perfect. I'm also not so clear on how this is much of a spam signal. The attackers get https certs, and the spammers pass DMARC. So BIMI is more aboout brand protection than spam resistance.
As well, there are 630MM domains registered and bimi radar only tracks 71MM. It only tracks 92 "large public companies", not even say, the F500. So I don't even have any confidence in what they are reporting, even within the scope of what they are claiming.
> I had to back down from quarantine in the past due to the way some customers wanted to forward our mail (breaking SPF of course but also breaking DKIM, thus not forwardable with DMARC p=q).
They should implement ARC instead.
> Even without p=q there is still a signal and MUA (or with server assistance) can flag the mail as suss, so p=none is not "wrong" or even "inadequate".
I strongly disagree. It's an absolute menace when trying to protect against forgeries. Worse than no DMARC at all (green light is worse than no light etc.)
> As well, there are 630MM domains registered and bimi radar only tracks 71MM. It only tracks 92 "large public companies", not even say, the F500. So I don't even have any confidence in what they are reporting, even within the scope of what they are claiming.
You don't have to take all the domains to have a sufficiently accurate general statistics.
Thanks again. I wasn't even aware of ARC. I'll start recommending that. However, there's a big gap between what customers "should" do and what they are willing or even capable to do. I'm sure most people and most companies have no idea how email even works these days. Eg, I'm the only one on my team of 10 infra folks that has any clue at all.
We are just going to have to agree to disagree on p=none. Regardless of the policy, the receiving MTA and MUA has all the signal it needs. p=none is very far from being a green light.
> You don't have to take all the domains to have a sufficiently accurate general statistics.
I'm not a statistician, but I would think that without random sampling, you do? There's no description of how they selected which domains to track.
I believe this about Google wanting to see all emails. When I use my own email server (and I do), Google don't get to peek at my email when I communicate with people that are not on gmail. They don't get to see what's in the mail, and they don't get the meta-data like who I'm communication with.
This will stop only if/when EU gives Google an appropriate fine of a few billion $$$.
Absolutely. And also its own users. E.g. my gmail account is "locked". Even though i know the password. And a secret answer. And everything else. G just doesn't let me. Because I switched jobs and don't have previous IP address anymore.
i blame our reliance on email addresses as usernames for services. i don’t want multiple email addresses, and the idea of switching seems overwhelming.
hot take: grumpy techies and admins killed email already 20 years ago. They were so intent on fighting html mail, top posting, and other perceived abhorrations that they forgot to make sure that it actually stayed alive, evolving viable platform for development. There were all sorts of cultural issues that made it so that there wasn't any major active promoters/evangelists/advocates for email pushing it forward, and so it has been rotting away for the past decades.
Hey.com solved the problem pretty easily: just have a positive list of email addresses that reach your inbox and the rest are sent to a screener folder.
It has hardly any spam added; that goes in the spam folder before it gets to the screener.
If you get attacked by an angry Internet mob (thousands of legit email addresses flooding you) you can have their security support clear them out for you, too. Pretty cool.
Finally, Betteridge's law of newspaper headlines is letting us down :)
I've had a bit of a email server for my family in 1999-2002 era, as a high school and university student. I've looked into it several times recently and it seems like the barrier to (effective, practical, reliable) entry is so much HIGHER than it used to be, unlike with almost all other technology.
Apologies for how long this is, but it's a fun piece of internet history. Back in the days of Slashdot almost every post about potential solutions to e-mail spam was responded to with the copypaste quoted in this post.
I think it's funny that we eventually got a technical/market based solution that was a result of gradual cooperation and centralization of e-mail control that required sacrifice of some of our e-mail freedoms (philosophical concessions).
It turns out that e-mail seems to have been a tragedy of the commons only capable of being solved by a regulating body and that as the regulating body functioned, people preferred it to libertarian e-mail.
The copypaste:
Your post advocates a
( ) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work.
(One or more of the following may apply to your particular idea, and it may
have other flaws which used to vary from state to state before a bad
federal law was passed.)
( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business
Specifically, your plan fails to account for
( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook
and the following philosophical objections may also apply:
( ) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough
Furthermore, this is what I think about you:
( ) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!
Doing the Right Thing should not be preempted by making a buck.
Maybe not even going to fully ML/AI but definitely some NLP to understand the intent behind the spam. It's sort of like the approach taken with https://spampatrol.io for forms. Probably can use it for email too.
regex is too specific, LLMs seem like they could help us make more general types if we can avoid hallucinations, maybe we could teach one to generate spam in order to teach another how to recognize it
More directly: spammers are killing independent email. Email's peer-node-trust story is so "version 1.0 Internet" that webmasters are left basically using heuristsics, shared models, and tea-leaves to determine whether arbitrary incoming messages should be trustworthy or not, and "they should not" is a good first-pass guess!
So Google (as the thousand-pound gorilla) is serving as a lightning-rod for a larger network-effect problem, which is "Users generally consider themselves better served if most unsolicited email they receive with no strong trust priors drops into a black hole." But that makes it very hard to be a newcomer who wants to establish trust priors.