I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
You don’t have to use blocklists to block messages.
I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
I don't use any kind of block list on my mail server, as I find the concept to be fundamentally flawed. I accept all incoming email as long as basic sanity on the connection is met and then apply bayesian filtering after the email has been accepted.
I get just about no spam at all (<10 per month, maybe).
Any guess on why people's experiences with spam are so diverse? I self-host and get fewer than 10 true spam emails per year (not counting marketing newsletters).
> Any guess on why people's experiences with spam are so diverse?
That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.
> > Any guess on why people's experiences with spam are so diverse?
> That would be an interesting research project.
Indeed. When you say get spam, do you mean pre or post-filtering?
Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the same time period there was only 1 spam email which was missed by spamprobe and made it into my inbox spool.
There's a fair amount of would-be spam that gets blocked during the SMTP transaction due to things like bogus host in HELO, etc. I don't keep any stats on those, I did at one time but it was too much noise.
My email isn't secret, it should be on every spammer list I'd guess. It has been the same since the mid 90s and is all over usenet, email list archives, websites, etc and I've never made any effort to mask it.
You mentioned receiving 10s of spam per day - are they being classified correctly and put in your spam folder or are you seeing them in your inboxes?
My mx hosted with hetzner also runs rspamd. Of the 32k mails received in the last month, 40% were rejected (postfix DISCARD, so the sender sees the mail as accepted but its sent to /dev/null - this only happens to mails scored very highly as spam, or sent to a spam trap address), 10% were greylisted and 1% were delivered to the spam folder.
So I'm also receiving 10s of spam per day, but they're all delivered to my spam folder with rare errors.
With the exception of an issue delivering to AT&T recently, I haven't had any outbound deliverability problems in a few years, but then again I don't send very much mail at all - perhaps I'd have more trouble if I did but most of my mail is incoming.
> are they being classified correctly and put in your spam folder
Mostly. There's probably 3-5 spams a day which get into my inbox; also 10-15 a day that wrongly end up in the "maybespam" folder (generally bulk that I'm not worried about seeing 100%.)
> That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
The old lists may be more plausible.
Some spammer once associated 'Robin Bennett' with my email addresses at some point 20 years ago and kept reselling it. Never used that name, didn't even know Bennett was a surname. I assume they linked me up with that name to fill up empty cells in their list and make it look more plausible.
It's a good way to filter spam. The last decade almost all spam calling me 'Bennett' is from some US political group, which mostly reminds me that it will be a long time before anything GDPR-like will pass in the US..
But almost all spam goes to that old 20 year old email address I don't use. My current email addresses are much cleaner even after 10+ year of use
> I thought self hosting is for aliases? Shouldn't those fix spam for good
Not quite sure I follow? Unless you mean "only allow emails to specific email addresses you've noted down", in which case, yeah, that works but also means a lot of admin when you want to use a new one (plus there's 10+ other people who use my servers for email, not just me.)
> unless you publish your address globally?
At least one of my email addresses has been published globally since ~1995. Others since ~2000.
> By lot of admin you mean that it's not automated? That's sad indeed.
How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?
Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to <randomword>.20.<knownaccount>@spamgourmet.com and you can't protect against those 20 spams.
But I don't find that burdensome at all, in fact I'd prefer that gmail let more stuff through to my spam folder instead of swallowing things that it misclassifies, because I have seen several instances of lost legitimate email, something I still don't understand how is deemed acceptable to the people who wrote it.
But I need to vote with my feet instead of complaining.
> One spam every three days is insanely high IMO (compared to what you will get on gmail)
That's funny since on gmail I get tons more spam than on any of my self-hosted domain addresses. What's worse, I see emails in gmail misfiled as spam when they are not, which is far worse. I don't ever see that happen in my self-hosted system.
And for most people, using a large mail provider is the necessary evil. I don't see much difference between a small mail provider using block lists and a large mail provider using the same block lists + their own proprietary lists and metrics.
Both result in other small mail providers getting blocked for arbitrary reasons with limited recourse.
Using large mail provider is not necessary evil in my book. It is just so much more convenient than self hosting.
Today, with ready-made containers and stuff we start having a fighting change, but the moat is possibly too big to fight with email falling out of favor
I think that gray listing is a great idea to filter out spam. Here's how it works: the very first time an email server get an email from a new IP it answers with a "temporary unavaiable" error. According to specs, a server should retry to deliver the email after a while, so a legitimate server will retry and it's IP will be put in a "withelist" (here in quotes because you can still do further processing of the emails to determine if it's good or bad). But a spammer will most likely not retry to send it, as their goal is to quickly send a big amount of mails.
I didn't try this personally as I'm not hosting my own emails (I tried but gave up soon), but I heard it works very well, no third party blocklist needed.
The trade off is that even legitimate mail that you are actively expecting (like an account confirmation or password reset mail) will be delayed by however long the sending server's retry interval is.
You also either need to apply the greylisting to some larger IP range (rspamd e.g. apparently uses /19 by default for IPv4) or otherwise specially handle some of the bigger mail providers, because some of them rotate through their servers between retries, so you could be in for a quite a long wait if you do per-individual-IP greylisting.
The biggest culprit I noticed this with was Amazon SES – a former mail provider of mine used per-individual-IP, non-configurable greylisting, and any mail sent through Amazon (which isn't just Amazon itself – quite a few companies are using Amazon SES for transactional mail and suchlike) would consequently almost always arrive several hours late (however randomly long it would take Amazon to finally re-use an IP during a subsequent retry attempt).
Even more infuriating, my mail provider's support would then claim that it wasn't their fault and they didn't know anything about any supposed greylisting.
Mostly. But then you get the "click the link in the email within 10 minutes" problem. There's also a non-zero number of "our mail didn't get through first attempt, oh well, give up" people. From running GL on my servers over a couple of years, it mildly cut down spam (on top of blocklists and fail2ban) but I'm now wavering over whether it's worth the hassle.
It cut spam tremendeously for me, but that's on an email address I've used and published openly since early 2000s.
Still, I've given up on it since plenty of email senders are not standard-abiding (they fail to retry), and I've kept losing email. I only caved in in the last 12 months after 15+ years of doing graylisting.
I have greylisting enabled for years. But recently some Russian spamers were able to circumvent those by using compliant SMTP servers. They even support proper SPF, DKIM, DMARC and all that stuff what you can think of.
That is the reason why I switched on some external block lists into the mix.
I wonder if anyone compared the effectiveness of postscreen vs greylisting. Can't find any
Both rely on filtering out non compliant senders, but postscreen's filtering might be less disruptive. Are there spammers out there who cannot pass a graylist but can pass postscreen ?
What we found 20+ years ago when we started to provide a very poor service to attempted spam delivery was that the spammers realised our servers were very 'slow' and stopped the attempts. We used to get 1000s of new IPV4 tuples / day, now we get < 10. If you put the inbound SMTP sessions to sleep this is wrecking the spammers bus. model, they need to deliver 1,000,000,000+ / day. Don't u fell sorry 4 them. The only time we don't tuck them in 4 a snooze is if they are in a 3rd party BL !!!
I don't follow. I am pointing out that using the same kinds of lists that the big providers use will result in smaller providers being arbitrarily blocked with little recourse. Do you disagree with that assessment?
First, yes. They (the small providers) have recourse, it’s just annoying. They voluntarily signed up for that annoyance because of their ideology.
But much more importantly, that question is orthogonal to my argument. Using blocklists is a good way to cut down on spam, the fact that it might block some trivial percentage of people who for ideological reasons might or might not be on those lists isn’t the receivers problem.
You want this to not be the case? Contribute meaningfully to solving the problem in a more effective way. Don’t blame the people who just don’t want the spam.
The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. That "trivial percentage of people" make up a substantial portion of service providers (for every Google there are many <100 user mail servers), which is what this thread was about.
I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
>The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. [...] I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
I don't think it's the tone. While the inability for senders to get off blocklists can be true, you're still not addressing why the cost to the sender to not be blocked should have higher priority than the cost to the receiver to avoid blocklists to make email admin more of a burden.
The gp you first replied to (PinguTS) is running a personal mailserver and resorted to using blocklists because reducing spam -- at the cost of some legit people not being able to send email to him -- is a tradeoff he's willing to make. You haven't convinced every user running mailservers that they should increase their spam burden because some small providers can't get off blocklists.
As another example in another communication channel... Here's a similar "blocklist" for cell phones that some users take advantage of: https://about.att.com/pages/cyberaware/ae/cp
Are "legitimate" phone numbers getting caught up in that block filter?!? Of course. But phone owners are trying to stop spam telemarketing calls. Telling them that some phone users are incorrectly on AT&T's list isn't going to convince cell users to quit using the block filter. They're willing to live with a few legit callers getting blocked.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
The correct response would be to set them on the blacklist for a time. People would complain and they would quickly change their behavior. Microsoft currently really wants everyone to have a Microsoft account.
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns
v=spf1 mx -all
This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam.
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
DKIM selectors aren't UUIDs. You can of course use a UUID as a selector, but you don't have to. My selectors are named S-YYYYMM (when I rotate the keys), so my current public key is at S-202001._domainkey.example.com.
A lot of tools generate UUIDs for the selector, just to get something unique without having to ask the user for something relevant or defining some other heuristic. For instance: the built-in helper tool for Zimbra generates a UUID by default, unless you provide something specific. I think a lot of people assume it should be a UUID just because they see UUIDs used in common examples.
Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
Counterpoint, our mail admins spend a lot of time trying to convince small-scale providers to shutdown the spam email coming from them. Lots of people who host at small scale providers don’t care about patches, so they send tons of spam.
Doesn't seem like a counterpoint to me. The provider you're describing isn't clueful. A clueful provider pays attention to what's happening on their network and knows how to make themselves an unattractive host for spammers.
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
> I have been hosting it on a bare metal Supermicro server in a proper datacenter, though.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
I've run my email server in VM's at a variety of ISP's like GoDaddy, Linode and AWS Lightsail and never had an issue, well once when Microsoft banned a group of IP's in a drag net. I filed a request with them and they fixed it in a week or so.
Real hardware servers are not required to host email.
I had tried it on a very inexpensive VPS before having my own physical server. The IP was blocked almost everywhere. Even though I filled out a lot of forms in Outlook, Yahoo, etc, it appeared that those major email providers had blacklisted the entire subnet of the VPS provider, and IP reputation seemed to have made little difference. Choosing the right provider is crucial!
A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.
Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.
Selecting the right host is very important. They should be unattractive as a host for spammers. Get this part right and you don't have to worry that tomorrow your neighbour is a spammer. The virtual/dedicated thing is not as important — many providers comingle virtual and dedicated servers in the same address space anyway.
One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.
I don't agree, but it is true that some aggressive DNS RBL may include Digital Ocean, for example, just because [1].
The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.
1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."
I don't think it matters which provider you choose. Big corps ban entire IP blocks, and you will get spammers as neighbors both at a low-cost provider and at a "proper datacenter".
FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.
It absolutely matters which provider you choose, assuming the provider is big enough to have their own address allocations (which is not that big, really — plenty of relatively small hosting providers all over the world have their own ASN and address allocations).
My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
You say that self-hosting a mail server is obvious and extremely low effort, which simply is not true. But ok, let's say that you happen to have a clean IP and no spam or dmarc issues. You still had to choose what to deploy, which is not obvious and actually setup/maintain your server which is not extremely low effort.
Please read the comment you are replying to again. I said that it's obvious that it's worth doing, not that the work to set it up is "obvious".
Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
I understood that sentence as something that can happen to you even if you believe it won't.
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
I agree, the core of the enterprise suite of Microsoft is still very much based on domain controllers and Exchange. This enables the needs for user management and access policy configuration. Where I live self-hosting with Exchange is still the usual way too. No serious company would ever use Gmail.
I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
I hope at some point the routine crushing of smaller providers gains the attention of the competition and markets authority, or its equivalents. It is crazy how Google and MS use their market power in this way and get away with it.
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?
G'Day Femto
Over the past 15+ years I have joined a number of groups like a web site 'web hosting talk' and all of them jumped all over my privacy and passed on my details to spammers / hackers.
I know what you are thinking -> how would he know that ?
Well its really simple I use DedicatedEmailAddressing ( DEA ) and our system tells me when a 3rd party tried to deliver a spam message using one of these DEAs.
Some of our customers charge suppliers for a new replacement DEA when they find the supplier has 'leaked' :)
Yeah, the host I use sells on webhostingtalk and I'm 99% sure they're selling my info on. My cPanel username there is the only place that I have that username, and that's what a massive chunk of my spam is directed to.
I'm a newbie to YC, can't c 'Edit' button, so I'm assuming it is like Twit. I'm not saying I wouldn't be interested, I think it is a really good idea, but really concerned that I don't find it is a waste of time after my privacy was again splattered
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
> Host your own MX servers but use someone else's SMTP servers.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer.
It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads.
The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
>I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
> I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
I remember in grad school, one of my classmates being upset about getting a porn spam email in Chinese and saying, “but don’t they know that I’m a girl?” I pointed out to her that I got the same emails (but had no idea what they were saying) and explaining to her that they send them to everybody.
It can very much depend on your domain. In the early 2000s I ran a domain for a firm and the domain had 'UT' in it. Well spammers thought it was related to university of Texas at some point and we went from around 10 messages a day to over quarter of a million. The immediate issue I has was poor back scatter protection so I had to configure that in a few days. That stopped 99.99% of the spam, but still caused massive issues with the mailboxes that did exist getting hundreds of messages per day even though we were blocking tens of thousands of mails per valid email addresses.
Spammers are miserable, I don't fault anyone for giving up.
SPF and DKIM no longer have any real value. Just look at SA and the values it assigns by default to mail with those headers. Couple that will the fact that spam now has valid SPF and DKIM - just makes them pointless.
DKIM and SPF, along with DMARC are more about authentication - they let you know that the message has come from where it says it does. This makes spoofing harder, not necessarily spam. Greylisting deals with most of the "spray and prey" spammers, though it does have it's own issues.
I do this too with a different provider whose whole business is relaying for self hosters. I have had exactly 0 problems since I started a couple of years ago. Its kind of perfect.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit.
Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Adding to the happy medium is to teach your friends and family to use Thunderbird so they can easily GPG encrypt [1] their emails keeping the nosey email providers off the email body. Also teach them to use the IMAPS (TLS) endpoint for their mail provider, usually port 993. There are probably simpler how-to's with pictures, I just do not have any of them handy.
It could be we have different circles of acquaintances. I have managed to get non technical friends to GPG encrypt their emails. I also talked 2 lawyers into using this and the two lawyers are not only non technical but have nearly zero patience.
I think you’d be very surprised at how much lawyers don’t know or care about any of that. They store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents).
> store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents)
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
They do. Most of them use proprietary https web interfaces that usually have "secure email" or "secure messaging" in the description but they are just fancy web portals. I despise those systems. The content is not encrypted at rest and can be leaked. With OpenGPG the emails are only decrypted on the recipients end points and can be deleted by request.
Not all of them. I keep seeing lawyers sending photos of sensitive documents using their cellphone and Whatsapp (before encryption), and most of them don't even care about all those documents still residing on their cellphone, at the mercy of whoever steals it since all the security they have in place is that swipe thing that anyone with good eyes spots in 2 seconds straight.
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
Perhaps you might care to share with us what types of devices they do have, seeing as my mental telepathy transgizmatron happens to be in the shop presently.
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
there is definitely software to help with pgp on ANY tablet, phone, or PC your friends and family have access to. The universal problem is no one cares to encrypt except a few computer nerds, spooks, and journalists.
It means you now have to worry about key management or else lose access to all your old mail, it possibly complicates logging in on new devices or via webmail, partially breaks server-side (spam) filtering, breaks server-side full-text search… so there are some definitive trade-offs there to be made.
Not a single member of my friends or family is going to use GPG encryption, not even my mother who loves me dearly. I had a computer geek "penpal" for a while and we did monthly emails to each once a month to catch up via gpg, and even that died off lol
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
To be pedantic, the version I linked to is a paid addin. However, you are correct - there is/are unofficial unpaid and unsupported versions available that I was unaware of, per the FAQ and the repository that you linked, which has nothing to do with GPGTools GmbH; from the repo, “This does not constitute any endorsement or promotion of our releases by the respective copyright or trademark holders.” Thanks for sharing this.
> You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
I've been hosting and operating my own MTA MX with Postfix since 2005. I have Postfix set to use "MailDir" format storage (one file per email) and then use Procmail filters to direct emails into per-sender or per-topic specific folders when they arrive on the server - nothing needed to be done in the email client. Dovecot provides the IMAP4 client interface. Thunderbird connects via IMAP4.
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
No...it isn't the only sane choice. If it works for you, great. Forunately for Internet freedom, there many of us small timers left who have no issues hosting our own email.
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
Indeed. I host my domains with dreamhost and it turns out that outgoing mail from their servers will get marked as spam by Google. The exact same mail sent through a gmail address (whether under gmail.com or a custom domain) will be delivered no problem (although after the sudden closing of legacy free email, I found that emails that I had been sending via gmail with my own domain that had been getting blocked by spam filters were being delivered when they came from a gmail.com address).
The problem is your outbound mail server does not have a good reputation. You also probably do not have a large enough swath of IP space to prevent bad neighbors from becoming your problem. Lots of tricks to getting your email sent reliably to most mail servers. What got me out of hosting mail was trying to send an email to a potential lead I met at a local meetup. His email was hosted at some very small and relatively unknown university, but my email was flat out rejected. Something about that moment just clicked that it's not worth my time to chase down the admins and resolve the problem. I'll just start shunting my email through O365. I still selfhost everything else I can, but I offload mail management to a provider
It's more than that. I'm on a mailing list for a social organization that I've belonged to for 20 years, and has been hosted at the same place for all of that time. I have marked hundreds of the messages not spam, and still Google sends about 80% of them to spam.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
As a counterpoint, I host a couple of domains with Dreamhost and have had no such issues. They've been serving the email for my main domain for 15 years and AFAIK I've never had any of those emails go directly to spam within Gmail or elsewhere. I do periodic smoke testing by sending test emails to different place, plus I have a reasonable number of external users making use of the domains for email as well. FWIW I moved all my DNS records to Cloudflare, but that was only in last couple of years.
Sure there is.. unless you mean no recourse in this environment. The natural recourse would be to force Google to divest GMail from it's core business.
No, the recourse is to force net neutrality. You must process all emails the same way and they must go through the same filters. In fact if spam processing is part of modern email it should be part of the standard and then all mail servers must be forced to conform to the standard.
I just randomly looked at 8 different emails in my inbox. All of them were from different email providers (except google which was there twice). There's hundreds or thousands of email providers you can chose from.
iphmx 1
google 2
kornet 1
linkedin 1
secureserver 1
amazonses 1
self hosted university email 1
True. But they are not calling for a completely open system. I like this proposal from the author.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
> I don't mind doing some paperwork or paying a fee to prove I'm legit.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
I can just imagine the headline. “Ask HN: Google sent my mail bond to charity for no reason and has torpedoed my small business, and I can’t get in touch with anyone to make it right”
I can imagine headlines like that too, but the idea of electing 5 (or some other odd number of) entities is that they would be able to share among themselves the cryptographically signed evidence of the spam they detected, and then the bond slashing would require a majority vote.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
You're describing one of the solutions made by Ironport, "Bonded Sender". Their solutions were sold to Return Path and Cisco later bought them out, presumably with the bonded sended solution still belonging to Return Path? [1,2]
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
Is there any actually money to be made in hosting email for people? I genuinely don’t know but my suspicion is that GMail, Yahoo, Outlook, et al are loss leaders for their owner companies. I suspect people at those companies would be quite happy if the protocol got unfucked enough that it small players could participate without negatively impacting the network.
Spam has been been fought for decades, you can rest assured any obvious solution has been tried and either doesn’t have the desired effect or is impossible to implement.
You ignore the fact that there are perverse incentives among the participants. It's possible to implement, and I'm doing it myself. If I had more time to spend on it, we could end spam. Instead I am fine as is: most of the spammers have given up.
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
signup confirm emails are not what i'm describing, because you need to establish and filter the initial offer that they send you via email itself, which is still prone to phishing.
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
This is useless because users are stupid, people sending the mail are stupid, people getting the mail are stupid, UI people creating interfaces are so stupid society could be improved by putting them in a box and mailing them all to some wasteland and hoping they form their own society there or starve.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
This makes sense for service emails and is similar to how push notification services like Pushbullet work, but can't work for humans. You need to be able to give your email to someone IRL so they can send you a message. Mutual approval would be possible in the "we just met and want to exchange emails" situation, but that too breaks when you legitimately want to give anyone the chance to message you.
> but that too breaks when you legitimately want to give anyone the chance to message you.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
I know, but anything out of band won't really work, because that can be phished even more, plus as described above, there's no real need for it either (IMHO).
That's essentially how www.hey.com works! I thought it would be tedious at first, but I don't mind it, and it's done a great job of making it so I only see what I want to see in my inbox.
From a quick read, hey.com still allows arbitrary people to message you and just initially puts them in "The Screener", which doesn't seem like quite the same thing as an absolutely "completely whitelist (opt-in to receive messages) basis".
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
Spam has been a thing forever. Literally forever. Yet people have been dealing with spam far better than the big boys, and doing so for decades.
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
I'd imagine they do this to not tip off spammers when a message goes through. It's the same idea behind returning a 404 when trying to access a resource you don't have permission to, or not telling users if an account actually exists under an email when doing a password reset.
It's simplistic for a spammer to tell if mail is getting delivered to end point. EG, just have a gmail account, and spam that account. Simple. Done.
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
A normal company could engage in these practices, but it is settled law that a monopolist may not. A monopolist may not use their monopoly in one are to supress activities in another area.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
Substitute “HTTP” with “SMTP” and you already have that today… SMTP in plain text isn’t generally used anymore, it’s always transported over TLS. Of course you can run whitelist-only, but how is that ever going to work?
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
Whitelist is easier. If I added an mail of a friend in this system, we can communicate. Everything else goes blackhole.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Of course whitelist is easier, but how valuable would your email be if you only allowed your friends to email you? Goodbye order confirmations, subscription reminders, recruiter emails, notifications…
Sounds like a dream, really. And you could always have it be user-driven action, like "Copy this line and paste it into your email provider's whitelist", framing it as a positive win for the user since they have total control over who is allowed to communicate with them.
I do want arbitrary people to reach out to me regarding my home page for example, so any just-allow-whitelisted-messages proposals are useless in that regard.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
Certs for MX servers are supposed to have the MX as subject or SAN, not your email domain. It's important when the sender enforces encryption with a valid cert (e.g. MTA-STS, or config in the mail server, or many hosted solutions like Google Workspace also support enforcing this for selected or all domains).
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
I believe you're correct. I should clarify that the SSL cert problems I have are only related to my client connecting to the server to send or receive messages.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
I have used Mailinabox on a Hetzner server for about an year. My email delivers to all the major providers. However, small providers will occasionally block my email. So I continue to use my Gmail address for now.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
For me, the issue wasn't that I had to fix it frequently, but that when I did it was urgent, stressful and disruptive. Eventually the VDS I was renting had a fatal HD crash and I gave up.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
There are many benefits to running your own server. The three biggies for me are:
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
Anything that comes in from an address I have never seen before is handled specially. But it's not hard to filter out the obvious spam. Just a handful of heuristics on the from and subject lines (e.g. if the sender's name contains common English words it's probably spam) takes care of >90% of the cold calls. The rest I just look through manually once a day or so.
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
The maximum bound for most gTLD registries is ten years (have a dig around the ICANN website if you'd like to check), not fifteen. Sure, there might be a ccTLD that allows you to register a domain for longer than that, but I'm not aware of any, and ccTLDs usually have lower maximum renewal/registration bounds than the gTLDs.
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
There is no such thing as a BCC header. That's the entire point of the BCC.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
Typically, if you sign up for an email account, you get an email address like skywal@gmail.com or skywal@yahoo.com. Alternatively, if you own/host skywal.com, you can have an email address like skywal@skywal.com served from a computer in your home.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
So in the example of the parent, he's got a domain name registered somewhere (mydomain.com) and he set, in its MX records, the gmail server. But how does gmail make the connection between that address and your gmail address then?
So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host skywall@skywall.example with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, add our servers as your MX record in your DNS, or transfer the whole domain to us and we'll do it for you". In this case we're doing the 'changing the MX records' part.
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for skywall@skywall.example". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a user@gmail.com address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
delv MX ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com. 295 IN MX 20 alt2.aspmx.l.google.com.
ycombinator.com. 295 IN MX 10 aspmx.l.google.com.
ycombinator.com. 295 IN MX 20 alt1.aspmx.l.google.com.
ycombinator.com. 295 IN MX 30 aspmx4.googlemail.com.
This is me using DELV to ask "where should I send email for ycombinator.com?" and I have four responses. Column 5 tells me the priority. Lower numbers are higher priority. Unsurprisingly, this is Google. But let's see where they host their DNS, shall we?
delv NS ycombinator.com @9.9.9.9
ycombinator.com. 159148 IN NS ns-225.awsdns-28.com.
ycombinator.com. 159148 IN NS ns-1914.awsdns-47.co.uk.
ycombinator.com. 159148 IN NS ns-1411.awsdns-48.org.
ycombinator.com. 159148 IN NS ns-556.awsdns-05.net.
So AWS. So they have separate DNS to Email, and could change those MX records to host their email anywhere else, without needing to change or move ycombinator.com's DNS from AWS.
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
There are some special domains of the format <reverse-ip-address>.in-addr.arpa and <reverse-ipv6-address-digits>.ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
> The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
So, mandatory third-party doctrine/warrantless surveillance.
It’s a misnomer to say you can “own” a domain name. You cannot. The better word would be “rent”. Because of that you should think about what could happen if your domain name registration lapses and someone else registers it.
I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.
I think people who get their email on their own domain generally consider it as their permanent home and don’t think about what will happen when the domain no longer is registered to them. Using the apartment rental analogy works up to a point. With an apartment you can get the postal service to forward your mail to a new address. With a domain name there is no way to get this to happen. Whoever holds the registration controls the MX record and can do whatever they want.
In case of death it's indeed an issue if someone isn't planning ahead.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
Most emails are still going through servers owned by Microsoft or Google, so what does self-hosting email accomplish in reality? Some government entity likely has warrant-less access to most of your emails regardless. I like email as a way to communicate, but speaking pragmatically, it’s just not a secure means of communication in 2022.
My server uses encrypted connections and my own certificate. If I send an email to a family member on my server, no one on the Internet can read its contents. If I add a recipient and FAANG or Microsoft then yes it wouldn't matter. If that is truly important, we can learn to encrypt emails locally first.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
what does self-hosting email accomplish in reality
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
Using a 3rd party email host that isn't Google has the same effect with a lot less effort. I switched to FastMail for this reason, even if Google claims not to scan paid Gmail accounts for marketing purposes.
I support the author but let me tell you a counterargument I don’t think he devotes enough to:
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
email spam is not a real issue. big uncooperative gigacorps fucking it up for everyone is.
responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
when the user clicks "spam" gmail already uses that to train their fancy AI and if you are not a small nobody, then they already alert you that ooops you sent something spammy via a feedback loop [1]. (see also how Mailchimp proudly claims they work with big integration partners like gmail ... https://mailchimp.com/help/how-mailchimp-prevents-and-handle... )
whitelist clearinghouses exist [2] but they are not terribly useful, because there's most of the signal to use for reputation is hidden :/
> responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
Are there any real-world examples of this? I know other decentralized networks like Tor and BitTorrent have some sort of reputation and feedback system. What type of "spam" do they deal with and how well do they deal with it? Are there any systems more similar to mail with mechanisms to prevent spam?
Spam is a serious issue, not just in email, not just in decentralized systems - it's one of the main issues in technology today. If there's a solution, even a proof-of-concept in a smaller system would be great
As a practitioner, I can tell you that this proposal would be quite a bit behind state of the art for email spam filter accuracy. For example, there's a surprising amount of legitimate mail that gets sent without authentication, and which you don't want to block because your end users will get mad if it goes missing.
I may also point out that "reputation scoring for netblocks" is the exact problem the original blog post was complaining about. He was trying to send from residential ISP and VPS netblocks that had poor reputation, and saw delivery problems as a result.
if netblock reputation were known small non-spam senders would congregate, this would help weed out the spammers.
but google et al. don't provide any feedback. nor any way to build reputation for your IP. (where are all the staking blockfoo chainbar solutions? or at least let people pay a one time fee for some exception)
anyway, since we already have certificate transparency, we could have a similar one to look up MTA/domain repu provider, then sending DKIM signed HTTP calls to the reputation report address, and let people aggregate it.
> "... small non-spam senders would congregate"
Congratulations, you've invented the email service provider. Most small non-spam senders don't want to spend the time figuring out IP reputation, DKIM, or SPF already, so they pay someone to do it for them. What makes netblock reputation different?
Reputation isn't public because then spammers game it and you get worse filtering outcomes. ISPs learned this the hard way.
Everything is getting harder as tech gets more and more complex. (As IT matures.) That's okay. The problem is that the incentives suck and there's too much business in email. The big ones have built their moats, and there's not much to do.
"SMTP" was simply never made to be resistant to this. (Nor the web in general.)
> And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
> Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
It is as good as it is exactly because of the requirements author finds tedious.
> It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
They do exist but there are a bunch of problems with that. People have varying definitions of spam, getting paid to whitelist someone creates a perverse incentive or not getting paid will quickly overwhelm the organisation.
Things could be better if we could enforce sender authentication (SPF/DKIM). It would assign a direct cost to getting your domain blacklisted. But if nobody is taking away rest of the spammer's domains (or keeps selling them new ones), they'll continue.
Spam is an issue, but it's not the one that impacts me the most. I rarely get true 'spam' in my inbox.
What impacts me the most is a never-ending problem with Gmail classifying messages as spam that were actually important to me. Time sensitive announcements of meetings or events, for example. Many are coming from senders I've been receiving email from for years.
Gmail is convinced that almost every technical-related email mailing list I'm on is a spam source, despite my constantly going into my spam folder and telling it dozens of messages from those mailing lists are not actually spam.
Meanwhile, what I do get is a barrage of promotional emails - what I consider very much to be spam - from corporations that at some point have had my email address and now email me multiple times a week. Those sail right past the spam filter into my "promotions" folder and accumulate...
Is Maddy's IMAP+object storage setup mature now? Last time I tried it I was testing against minio and trying to get my 25gb inbox synced over. The transfer was in b/s, would've taken forever. I love the project but haven't checked in on it for over a year now.
I use IMAP+on-disk SQLite, and make sure to take backups, I haven't run it purely off of object storage yet so can't comment on whether it's better with such a large inbox!
I use maddy + dovecot. maddy's IMAP implementation is good for small stuff (and trivial to set up!), but still performs very slow for few-GB inboxes or larger.
o/t: Could you add a close button to that ethical ads element? It's obscuring a decent chunk of the screen on mobile and I can't figure out how to dismiss it
The ecosystem can definitely use more work, but I've found maddy to be really good, because it's all in one, makes very reasonable choices, and is easy to configure.
I put maddy first because it's what I personally use (without dovecot) -- I make sure to back it up as well, and that's enough for me/most people, I think. It's the closest to the whole package which doesn't rely on.
Oh you know what, I completely forgot about iRedMail:
It was a huge mistake for email receivers to take on the cost of filtering spam. Of course given the evolution of the internet and email it is easy to see how that mistake happened. Nobody had a crystal ball. But the only solution here is to raise the cost of sending email to the point where spam is no longer profitable.
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
This indeed looks like a good direction. A decade ago Freenet (not sure if it still exists?) had a problem with spam on its equivalent of USENET. It was pretty bad, until they changed the protocol so that it's the sending node that keeps the message, which is then pulled (or not) by the recipients. It made a lot of sense to me: I'm the one sending you the message, I want you to see it, while you don't even know whether you want to get that message. So it's me that should pay for storage and propagation of the message in terms of bandwidth and disk space. Not sure how it turned out in the end, but the approach seems right to me. It's like phone calls: it's the caller that pays the cost, not the one receiving the call.
Wow that's ingenious - and obvious when you think about it. I guess evolving the email standard is a much bigger obstacle though, no matter how clever the proposal.
it's ridiculously easy to create a template that is stored and when the remote client pulls the message only a few variables has to be substituted. it's how spam is generated after all :)
Yes, but in the context of Freenet specifically, there's a lot of encryption and hashing involved in every transaction, which are computationally expensive, and there's no CGI equivalent at all (the whole network is basically static, append-only content addressed with the content's hash). So to create many messages you'd need to first generate the content, then encrypt and hash it, then insert it into the network starting at your node, then provide signature and checksum verification on every request. After some time, if the content is widely requested, some other nodes will cache it, but you can't really expect spam to propagate via web of trust; it's more probable that it will be added to spam filters which, themselves, will propagate. As everything is signed, you'd need to create a fresh node for each spam, but that means your new identity is not trusted by anyone - you'd need to convince at least some users that you're a user that won't abuse the system. Only to get your whole identity blacklisted once you do abuse it.
It wasn't perfect, and Freenet's usage was a huge PITA from the usability/ux perspective, but in a constrained environment like that this change in the protocol had much bigger impact than it would on the open web. Not sure how it turned out, but the trend until I left the community was very positive with regards to spam.
Something different: a hash which is expensive to calculate but cheap to verify. E.g. calculating a string of bytes to append to a hash stream on order to produce a hash with a certain number of leading zeros; you provide the hash and the bytes, and it's trivial to verify.
That's almost exactly my off the cuff suggestion. Of course my brain is seeded by both bitcoin (finding a hash as proof of work) and HLL (which counts leading hash digits to estimate set cardinality).
Like cryptocurrency this will be a moving target. You would need a difficulty setting - but where as cryptocurrency only has to track one thing "total hash rate" this would need to track two things "minimum supported hardware for legit sender" and "hardware threshold for attacker to be successful with a mining rig they can afford, based on the income from the spam".
Each email provider might come up with different difficulty levels based on what they thing this is. So some handshaking might be required. And less computer literate people would be stressed why their email is taking 6 minutes to send. I think it would be hard to implement.
Oh I know. The point is something hard that the sender needs to do. Ideally it wouldn't waste lots of power, but it must consume some scarce resource, or else it wouldn't cost anything, and the cost should be borne by the sender, so e.g. a RAM-intensive hash function wouldn't be right.
That is a clever idea but I think it'll still fail so long as email (SMTP) is a fire-and-forget architecture. As long as you have that asymmetry, your SNR is going to suck.
If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways. This could be something just like DKIM / DMARC / SPF, or even something more exotic, like making the client do proof-of-work with difficulty tied to how suspicious that client is to the server, and also the delivery scope/scale. Or forcing the client to wait for ACK for valid delivery while slow-walking it.
This gets around some of the issues in cousin comments, with respect to punishing botnets and rewarding lawful players. Established, high-trust players pay no cost. Suspicious players can still get through, albeit with a tax (that should be trivial for low-volume personal MX, but expensive for high-volume spam). Furthermore, it's adaptable.
> If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways.
That's basically what graylisting aims to achieve.
Yeah, this is essentially a form of greylisting. The difference is (as I understand it, this is fairly outside my domain), with the current setup, MTAs can accept an email, and it ends up getting blackhole'd or spam-folder'd anyways. My hypothetical scheme would put more onus on the first "boundary node" to report on errors/compliance. Basically the MTA tells the client what hoops to jump through, and the client gets some indication what will happen once those conditions are met.
That could be an exchange like: "Sign this nonce, and your message will be vetted", or "this is very suss, you have to do X difficulty hashes to have any chance of delivery, and regardless it'll be flagged as potential spam". Or perhaps just a guarantee on how an action would affects the message's "spam score".
This could be used alongside nested packets/envelopes and various headers/trust levels in a network of trust to give a message some overall trust level.
The problem to adding a cost to email is that it affects everyone. The amount of CPU power you need to waste to make most spam not viable is so much that it isn't worth it.
It definitely does not - you can allow different work loads for different senders. Mailing lists you actually want can be dropped to zero for instance.
Most spam comes from new address pairs, not existing ones. Requiring high cost to get past a first-contact filter, then near zero forever after, is completely reasonable and would practically eliminate unsolicited spam.
But now the sender needs to know the receivers policy and if they remember that there has been contact before. Or I guess you change SMTP but we still allow unencrypted connections so good luck with that.
For newsletter style stuff, nah. The "confirm your registration" email can "pay" to get past the wall, and then you're done - approved pair established, future letters can probably be zero cost and everyone receives the same one.
I wholly admit that this is arguing theoretical setups and that's always problematic, but of course patterns would be established pretty quickly. There are loads of simple tactics that would still make spam dramatically harder, and legitimate use nearly unaffected. The current reputation system has clear, massive gaps that really don't need to exist.
The address 929@homeaffairs.gov.au, which must be used by Australian permanent residents to update their personal details, refuses to accept email messages unless they are from big tech.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail
system has been rejected due to the sending MTA's poor reputation. If you
believe that this failure is in error, please contact the intended
recipient via alternate means.
929 is the form. The email address is for completed copies of the form.
Permanent residents are a class of non citizens who have the right to permanently reside in Australia. Most cannot vote.
All of it can be done online through their portal. The Australian government spies on everyone anyway so using the portal presents no increased attack surface for anyone who would care.
Really it's there as a legacy technology. Most people under 35 would certainly use a portal. Probably most of them over 35 as well at this point.
If you used an immigration lawyer to apply for permanent residency, you can't use the portal to make those updates, because the original visa is not associated to your portal account.
So your only other option besides emailing the form, is to print it and bring it in person to an office.
I know this, because this was my situation until recently.
A few months ago I changed my PR subclass and the new application was done using the portal. Which means that I can finally use the portal to update my details.
E-mail is complicated, sure. But I’ve had it up to here with people who give up running their own server and then go on to vastly exaggerate how infeasible it is, in order to placate their own conscience. It’s not that they’ve gotten tired of doing it, oh no; (they say,) it’s entirely the fault of Google, Microsoft, etc. who’ve made it literally impossible to run your own e-mail server. Except it’s not impossible – lots of us do it, still. And now there’s one fewer of us, so the rest of us have to work that much harder when the next monopolizing standard comes along (BIMI, anyone?). Sure, you don’t owe us anything, but thanks for nothing when making these public rants; you are scaring away people who might still be inclined to help!
There's alot that goes into a mail server stack, but it's no more complicated than k8s or other stacks these days. My preferred setup is rspamd/postfix/dovecot/roundcube. The docs are good and the mailing lists are active & archived for easy searching
For a pre-packaged mailserver environment, take a look at mailcow or mailinabox
There's the catch - VMCs are not a mandatory part of BIMI. Though if you want to establish trust, someone has to be willing to put their name on the line and verify everything required. If you have a better approach in mind, I'm sure a lot of people would love to know.
Trust (by way of VMC) is the whole point of BIMI; it’s what’s in all the marketing copy: The fact that nobody else can send mail with your logo. If you merely wanted to send e-mail with your icon on it, that already existed: the X-Face header has been around for decades.
Absolutely not, BIMI is a way to provide brand identity for recognition/marketing purposes. Trust or security is not the standard's goal. Being able to trust the logo is an optional extra.
You are free to read the BIMI standard's section 2.1 containing its high-level goals.
The next section also literally says "This document does not cover the different verification and reputation mechanisms available, but BIMI relies upon them to be in deployed in order to control abuse." It's not a standard meant for establishing trust, it does not mandate requiring a VMC.
Just like with e-mail, it doesn’t matter what’s in the standard, what matters is what the big providers actually do. If Google (Gmail), Microsoft, etc. will simply show any BIMI logo without VMC verification (which will never happen), then I will concede that BIMI is not a monopolizing standard. It’ll just be a tracking pixel.
Showing any BIMI logo is an absolutely unreasonable thing to demand from a large-scale BIMI implementation. It does not make it "monopolizing", I don't think you even know what the word means.
When the big providers all require VMC to show BIMI, then VMC is not optional, no matter what the spec says. Claiming it is optional is then disingenuous.
As I said in the linked post, logo verification is not a problem which can be solved. Identical trademarks can legitimately be issued in different fields, and both still be valid. Let’s say you are a brick manufacturer, and have paid an arm and a leg to a VMC certificate authority (previously a HTTPS EV certificate authority) for your logo, a nice iconic square logo. Then someone else can simply come along, register a flower shop in another country, use a different VMC issuer and get an identical logo issued to them. They can now send e-mail invoices to your customers with your logo on it, legitimately obtained, and the BIMI system will have trained your customers to trust your logo.
Any fix for this you try to implement will make the system even less usable for its stated purpose, or more suited to only large players and unusable in practice for smaller operators.
> When the big providers all require VMC to show BIMI, then VMC is not optional, no matter what the spec says. Claiming it is optional is then disingenuous.
What do you mean "no matter what the spec says", it is the spec we're talking about. It is what you argued against several times.
If you had started with saying "big providers' implementations of BIMI", then it wouldn't be wrong to say it's required but it's still not "monopolizing". Requiring you to prove your claims using a third unrelated party is simply not that.
> As I said in the linked post, logo verification is not a problem which can be solved. [...] and the BIMI system will have trained your customers to trust your logo.
There are caveats to each system. It does not mean the problem is not solvable to a large extent.
Secondly, it's pretty clear who to jail for the attack described. I'd say it's even a positive side of the system if that's the type of attacks we'd get.
> Any fix for this you try to implement will make the system even less usable for its stated purpose, or more suited to only large players and unusable in practice for smaller operators.
That's simply not true. The price of a VMC is really not that high for any business that doesn't only employ one man and his dog.
Still relevant, since we were discussing email, it would not need the whole "internet community" to agree, just a handful or two of provider could impose it in practice.
Standard are useless if majors providers apply a different de facto norm. That behavior has a name, it is called a cartel. And on some matters, that is punishable by law.
Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.
All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)
My fear is that something similar will slowly happen to everything "compute". How long before my bank's website won't let me login if I don't use a computer with secure boot and a browser installed from an app store? McDonalds app on Android won't run on a de-googled or rooted device... At this point one may argue that there will always be computers that you can compile and install your own Linux. Yes, that is true. But just like I am not likely to have un-googled andorid for some apps and googled one for others, the same way it won't be practical to have one computer for some apps and the other one for others. And the one that will win will be the one that lets you login into your bank account, for simple and practical reasons.
I agree with the pains, but the options are not juts Big Tech or self-hosting. There's a myriad of not-big-tech email providers out there, for example there's Posteo, who use open source software and green energy. They are going strong for 13 years now with 400+k accounts.
Sending email out is a royal pain. Trying to deal with a Microsoft ban on my IP even though it’s sparkling clean for several years. DKIM, DMARC, SPF etc all ser up, reverse dns, you name it. Looks like Linode is being blocked as a whole pretty much?
Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.
I've been running a private mailserver for myself and a couple of friends since 2002. I hit a similar issue with Microsoft last year.
I route outgoing mail to hotmail.com, live.com, outlook.com and msn.com domains via email-smtp.us-east-2.amazonaws.com. I started doing this when I noticed a large portion of e-commerce email I was receiving originated from amazon vms and they clearly had no deliverability issues.
The ability to do this is a paid-for service, and needs to be configured appropriately. I was expecting to have to pay for a VM when I set out to do this. However, it turns out you do not need that - you can just configure the SMTP forwarding without needing to purchase any other products, and the cost for this is a few cents per 10k emails, which I can see would be significant for commercial services and/or spammers, but for a private mail server essentially means I will never have to pay. Amazon go to great pains to maintain deliverability for those servers, so you don't have to.
I don't really see a problem with adding one extra SMTP hop to outgoing emails tbh, but as no-one other than Microsoft has (to date) a permanent IP block ban with no recourse whatsoever policy, while I've had transient issues with most other big providers at one time or another, those four Microsoft domains remain the only destination I have to route this way.
office 365 hosts a bunch of domains though. My bounce was to a .berlin domain hosted there. Yes, I could route all smtp via a ESP and pay for it, but that’s centralization.
Interesting - whatever permaban it is that my IP block got didn't affect office 365 hosted services (verified with several different domains at the time) - just literally the domains I mention. Guess I've lucked out so far.
Wouldn't surprise me if Linode were entirely blocked. Also anything in M247 Ltd's ASN. They host a lot of VPN endpoints, including Mullvad's.
Step 1, I'd move off Linode. Find a local DC or business you can support by hosting a VPS or dedi box with them. LowEndBox might be an interesting place to search but avoid anything too famous.
Step 2, join this https://sendersupport.olc.protection.outlook.com/pm/services.... They don't actually email me when they block my IP, but at least when I contact them I can argue I'm already in their program and they didn't actually notify me of any sending issues. I've got myself unblocked relatively quickly this way.
IMO, any low-cost VPS provider is going to have a poor reputation. I've even had trouble running websites on some of them, as some "endpoint security" products have their IP addresses blacklisted, causing users to get alarming warnings if they try to visit sites hosted there.
You'll run into the same issues with the "free/trial" tiers of bulk email services like Mailgun.
I agree with OP, you pretty much have to at least send your outgoing email via an established, widely-accepted SMTP service provider, or in some other way pay a lot for a "clean" reputation.
I'd still say it depends. I've used memset (www.memset.com) in the UK in the past without issue. They have low end VPS offerings, but they're not as well known as say Linode.
It isn't so easy to categorise such a service by pure ASN, as they also offer dedicated and colocation services, so you might be blocking a licensed 'on prem' exchange box for a local UK council.
I have my personal e-mail on linode for probably 15 years or so and haven't had any issues with e-mail delivery (knock on wood). Had the same two static IPs there for years, and have always had things locked down so I couldn't be used as a relay for spammers.
mine is maybe 10 years? I don’t recall. No issues so far, but started seeing those Microsoft blocks when I moved my brother’s domain over to my server. I hardly send email out, so it could be unrelated to his domain, just that he sends much more email out.
Thanks I’ll check it out. The bounce link was to an office 365 spam delist portal. My request was auromatically rejected and my only option was to escalate it to support (one click luckily). I’m yet to hear back, but from the stories on the link I shared, I doubt it will do anything.
This isn't actually that hard to fix, it's just that for whatever reason, we seem to frequently have this blindspot that we don't seem to have in other industries.
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
> like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
and your comment seems to be saying it’s OK if we lose the ability to cook hamburgers at home, because there are other (more ethical) restaurants that aren’t McDonalds. am i misunderstanding?
The metaphor is kinda good. It's okay for people to refuse your home-cooked burgers because they don't trust them to be safe. There are actual small food vendors that comply with sanitation rules and you can have them provide catering services, professionally. And that is okay.
we mandate public health measures because it's kind of easy to check for "safe to eat" quality.
also arguably it's too ineffective considering how hard it is to open a restaurant. (because instead of harassing random taco trucks we should focus on other preventative measures, better visibility of food safety, better reporting and tracking)
sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
we already have certificate transparency logs, OCSP, CAA records, and all the fancy stuff, what's needed is something similar.
> sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
> what's needed is something similar.
There are more than a few out there. No offense but if you would have had dealt with spam, you would have known that already.
Another way to look at it: In what world are you more likely to be able to cook your own burger: One where it's literally just McDonalds who can now control how people talk about burgers, or one where there's also other restaurants as well.
I'll take a shot here. I'm not sure why you think Gmail or the other biggies can actually guarantee delivery any better or worse than anyone else. In a technical sense, they definitely can't.
In an anecdotal sense, I mean, I've dealt with my hostdime stuff, gmail and outlook. My stuff and gmail deliver just fine. Outlook does fail.
In a practical sense, see issue above. I think "non-delivery" via the big-boys getting things wrong on spam (and worse future outcomes, pretty easy to imagine gmail trying to "amp" or something similar email such that it cuts out little guys) is a far far greater danger or problem than an individual or smaller outfit tinkering with their own servers and occasionally getting things wrong.
No. To beat up the metaphor, I'm saying if we support the restaurants that aren't McDonalds and that are more "mom and pop" (and get others on board) then we can beat McDonalds or at least live in harmony with them.
To colinsane's point, I don't want to be a restaurant, I want to cook my burger at home. A lot of us do. You're OK with the Hostdime restaurant. I'm OK cooking my own burger. A lot of us are and we don't want to lose that right.
Said it above; but I 100% agree with you. My point has "second-order" effects.
I'm not necessarily saying your email has to go away and you HAVE to use hostdime. I'm saying the goal of McDonalds is to dominate and obsucre everything and anything that cuts into that is useful, especially third-parties like Hostdime. For someone like me that finds it too hard, but also finds gmail unacceptable this is a good (and should be obvious) solution.
How I explain this in real life:
"You'd be stupid to run a business off of gmail. When your phone or internet or electricity goes down, you can call a human and yell at them because you pay them. What happens for google? This is why I pay someone."
I am not a Google employee, but I do work with email anti-spam at scale. There's a lot to critique here but it boils down to three points:
1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.
2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.
IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.
3. The filtering behavior reported here is either misunderstood or misrepresented.
First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.
So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.
> The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
You say you're knowledgable, but I also ran my own email server for a couple of decades, and gave up for precisely this reason. Maybe this is even true, but when you're trying to get mail sent, YOU NEED THE MAIL SENT, and you can hardly wait 2 weeks or a month. Besides, your email server is going to give up and report it as undeliverable by that time. So you have to do the legwork to get off the block regardless of how long the "bigs" have the timeout set to. And that's happens so often, and is so onerous, it's not worth the hassle any more. Like many others here, I could have written this article.
The overwhelming majority of legitimate mail will never see anything remotely near a 14-day block. Again, it's reserved for severe, ongoing, high-volume spam scenarios; ISPs use other methods like short-term deferrals for more common/less severe problems.
If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else.
As a rule of thumb, email-specific service providers tend to do a better job of managing IP range reputation than more general purpose providers like VPSes.
The overwhelming majority of my legitimate mail has seen consistent and irrecoverable 14 day blocks for the last decade+. One on an IP that only did my email for over 6 years. Yet the oligopoly started to block blocks of IPs, including my IP, without recourse.
If you say this problem doesn't exist, you are either not familiar with the problem, misunderstanding it, or simply lying. I hope it is just miscommunication.
Like many here, I gave up self hosting email. To the point that I buy an SMTP service from one of the oligopoly. The incoming mail is handled fine by my server still.
Exceptions to the rule do happen; I'm sorry your experience was the exception. Dealing with that kind of thing can be quite a pain if you're not experienced with it.
But again:
"If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else."
You know why? Because any IP-adress that I can afford, is going to be blackholed.
You make it sound like the blame lies with the Linodes, Digital oceans or even the Hezners or ISPs. This is not their fault. The blame lies, entirely, with Microsoft and Google (And to lesser extend Yahoo) using a cannon to shoot a mosquito.
Again: My IP (the address, not the range) was fine. It had been fine for many years. Why then, must Google and/or Microsoft, randomly, block this address? Why can't they make exceptions for reputable addresses within a range of bad ones? (I know why: they are lazy and use the easy path: just block everything and accept some "collateral damage", especially when that "collateral damage" cements their oligopoli a bit more, and when avoiding that collateral damage not only costs more work, but enables competition to exist (and grow))
> IP range reputation is not easy to assess, sometimes even for email professionals.
The article is about a man who has been sending emails from the same address for the last couple of decades, and it has always been in his control. Is it difficult to assess the reputation of his address? You seem to be deflecting some blame onto the victim of the shortcomings of your system.
This is not my system, to be clear, but you might be surprised how little 20 years of data matters to filter accuracy, empirically speaking; recent sending behavior carries orders of magnitude more weight.
Have around 100 users on my self-hosted mailserver. Works alright for the most part. Once or twice a year, there are connection issues to small companies with weird settings. I just route those over an external ESP.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
The owner is from the lowendtalk.com hosting forums.
It was a side project for a few years and has been full time for last couple of years now.
He's very hands on. Has very strict rules about spam - both from sending and receiving perspective.
When he brings in new ip ranges, he babysits them by slowly warming them up until they have a good enough reputation to be accepted by major providers.
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
This thread is quite enlightening. It seems the engineering community predominantly would like to be able to self host email (and presumably other services). By proxy of that I guess that email hosting is not just for one user but maybe also friends and family or some community that you are part of. It's quite clear that managed centralised services accelerated the adoption of the internet, email, etc and the majority of consumers don't really care about self hosting but it also speaks to the fact that we've given up ownership of this aspect of our lives without clear understanding of how it will affect us in the long term. If no one but large corporations own all the services we use then it puts us in a pretty precarious position.
> ... [They use] spam as a scapegoat to nerf deliverability and stifle competition.
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
> This is the same Big tech story. They want to cut cost, you want a human touch. You can see a similar story here in HN every week.
Dang, sounds like our monopolized tech dystopia is probably inevitable. If only there was some state level mechanism to help balance the interests of industries maximizing profits and those of users who need a robust system which doesn't harm them.
There is a really great bit from Robert Reich (labor admin from the Clinton years) about how if you're business would collapse if you need to pay people more than minimum wage, you shouldn't be in business.
If your business suddenly fails as soon as you need to... support your customers, then you probably shouldn't be in business and only exist as a fluke in the current economic model.
The topic often comes up. Can't say I share the experience. My servers have never been put on a blacklist in the 7 years they've been running, and one of them operates from my residential DSL connection. Standard postfix+dovecot stack on an Archlinux VPS, I log in once a year to update the packages and make sure there is enough disk space left.
Essentially all residential IPs are blacklisted for email servers, at least on port 25. Many ISPs blackhole any traffic on port 25 to residential IPs. Do you have port 25 working on your residential connection?
Isn’t port 25 used for unencrypted traffic as opposed to port 465? I’m pretty sure I had a working mail server back in 2012 behind an ISP that blocked port 25.
Know what will kill self-hosted email? Giving up and hosting your email with the big guys, the less self hosted email servers there are the less the big guys feel any requirement to support it
All that "security" just to fight spam. IIRC it was estimated that globally spammers make $300M per year from their spam. It doesn't seem like much. Somebody joked that it would be better if we just paid them that much to do nothing.
I think it is different. With Piracy people want something (product/service/media). The quote from Gabe is about the fact that they are willing to pay for it, if it is convenient and affordable.
Nobody wants spam or the things promoted in the spam. The companies want attention from the users. I think the better version is targeted ads on Facebook, etc. Maybe that’s the closest analogy to Gabe’s quote.
> Nobody wants spam or the things promoted in the spam.
Lots of people want Viagra without having to see a doctor. There definitely is a market for it, and now there are companies that do provide it, I wonder if this will cause spam to shift to other products.
That's true! I had considered those to be scams rather than "normal" spam. When I look at what my filter catches, it broadly falls into three categories: scams, viagra/dating/sex-related products, and every day products ("buy this great ladder for your garage"). Too bad you usually can't analyze how many clicks and sales they generate.
The underlying problem is that a sufficiently motivated spammer can target tens or hundreds of millions of people with their spam without too much effort.
As a result, every possible scam and spam with even the slightest possibility of converting 0.00001% of recipients can now be a viable spam campaign.
The underlying problem is that it’s so easy to scale spam to a lot of targets.
Yup. Just requesting the sender to solve some riddle (and waste their energy in process) would turn the tables completely because the cost of sending would be non-zero. Unfortunately it would also mean that we would be sacrificing our planet again. But maybe the difficulty of the challenge could adapt according to some trust score?
I think the deeper problem is sending an email (or a billion emails) triggers someone else's servers to do most of the work. Email is beautifully cooperative computing, but that means it can easily be taken advantage of.
Spam is just the opposite, it is actually a pricing issue and not a service issue.
As long as the expected value per message of spam sent is positive, someone in the world will send as many messages as they are able to. You either fix this by raising their costs so that spam no longer has a positive expected value, remove their ability to send an unlimited number of messages, or both.
This is not exclusive to the e-mail system, robocalls are still a major annoyance, and the global telephone system is much more regulated. Mobile phones now automatically filter calls, even! It's wild.
Steam and Netflix made it possible for people to play/watch media more easily than going to the pirate bay.
What are spammers trying to accomplish, that a better product would prevent them from using email (that isn't just them shitting up the other service the way they shit up email)?
> Somebody joked that it would be better if we just paid them that much to do nothing.
I know this is in jest, but in economics there's this concept called "induced demand" that comes to mind.
"Public extortion" would be an interesting challenge, as it would be difficult to solve the problem of "Hey why don't you also pay ME to do nothing too?"
I am still doing it, on a cheap VPS no less. Yes, it is hard, and yes, some large email vendors drop my emails for no reason. However, if everyone throws in the towel, they (the monopolies) won.
I relate to this. I also stopped hosting my own mail server for this exact reason.
However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.
If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.
One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.
The paradox here is the same one patio11 discussed in "The optimal amount of fraud is non-zero", on the front page yesterday [0]. The more non-tech people have an email address, the more we have to prevent fraudulent email, and the harder it becomes to run your own email address.
The original email users were much more savvy and needed less protecting against fraud. Now my grandma has an email, and if we're not careful she ends up on the phone with "Microsoft customer support" giving them full access to her computer. Spam filters aren't just a question of irritation anymore, people's life savings are at risk.
I actually would rephrase it as there weren’t enough users with money to make using email a worthwhile scam medium.
Email is low trust and almost any user is susceptible to being scammed, because there aren’t good trust markers in emails, and companies use it in ways that make it indistinguishable from spam.
I think both are true. The kinds of scams my grandparents (and even parents) fall for are trivially recognizable as scams to me. It takes more work to defraud someone who knows more about the way thing are supposed to work.
But yes, it's definitely still possible for anyone to fall for more sophisticated scams, and there being more money to be had is a huge part of it. Either way the effect is the same: more protection is necessary than was before.
Oddly enough I do not have these problems after self-hosting for close to 27 years now (i.e. from before spam became a problem). I hardly ever get any spam and my messages seem to arrive where they need to be even when that destination lies in Microsoft- or Google-land. I have the usual assortment of DKIM/SPF configured for my domain, I send mail through a smart host operated by my IAP (at no extra charge) but for the rest I do not do anything special. Am I the exception to the rule, am I just lucky that my IAP's smart host has not been blacklisted by the likes of Microsoft and Google or is the perception of self-hosting mail to be fraught with problems erroneous? I suspect the latter to be true, self-hosting is neither difficult nor bound to fail just as long as a) you have some good spam filters (easy), b) your MTA is set up with the correct SPF and DKIM records (also easy) and c) you send outgoing mail through a smart host (easy to configure once you've found one).
I tried to operate my own email server as well on a VPS, and I have been thinking that the way to solve these problems is to solve the problem of spam itself. Detecting spam puts the costs on the email providers... when the costs should be born by spammers.
Perhaps some sort of digital stamp (digital signatures similar to stamps on physical envelopes) for each email sent paid for with micropayments in a cryptocurrency like nano (note: I don't own any crypto). Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers. SMTP servers should put all incoming unsigned emails into spam folders. This will disincentivize spamming (probably not eliminate it) enough that self-hosting emails might be possible again without having to swim against the tide.
Paper mail, phone calls, and text messages all cost money, and yet you get massive amounts of spam from each of those. No, putting a price on those does NOT help. What will happen is that one source of low-quality spam that will be priced out of existence will subside, and now companies with more money will be priced into the game, because they will decide that sending spam is competitive again, as there's so much less of it nowadays. So then those companies are sending you spam and there's no escape from that. What's the next step? How do you fix that? The "digital stamp" thing hasn't been thought out too far into the future.
Those are the equivalent of your banner ads and pre-roll ads. The next step is ad blocking. Since there's not a lot of companies that actually want to pay for this stuff now, there's actually not so many of them, and you can enumerate them. They'll try to randomize the text using auto-gen and eventually things like GPT-3. This might be successful. How do you protect against that?
Let's say you protected against that. The next step is ad integration. The kind of stuff SponsorBlock removes from YouTube videos. Small mentions of ads and sponsors, integrated into the content. Interaction reminders. Donation begging. SponsorBlock works well because all that stuff is public, but that might not work well for email. Are you willing to let an equivalent of SponsorBlock read your email? Would you trust it? It would require a completely new paradigm for such blocking addons, where we're sure - by means of technological assurances - that such a blocking program cannot spy on us by leaking emails back to the mothership. That's a tough one, and I have a feeling in the current browser-centric environment the effort is just so large, and the required approaches are so far from what's being done nowadays, and the payoff of satisfying a fringe mail-midroll-ad-blocking need is so small, that it simply won't be done.
What's next after that? SponsorBlock hasn't caught up widely enough to cause people to find workarounds. I don't know, but I'm sure it'll come.
> Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers
Given how much is spent on ads, I’d say that an email is still worth more than that. So, what you get is spammers paying minuscole amount of money to skip the spam filter and killing the idea at the root.
Or they can simply steal these credentials from one of the millions of hacked sites and cause additional trouble to them.
Within the confines of "a digital stamp is a good idea" [1], what you're saying isn't quite true: instead of being treated as a whitelist, a digital stamp can just be seen as an finite increase in credibility of some quantum that's decided over time by adaptive filters.
..or imagine if google just shuts down your account. Gives you no reason and suddenly stuff on your phone, your email and all those related services stop.
The issue with greylisting happens when you receive a lot of email from Google or Outlook servers. Different one each time, unless you're whitelisting them all - which defeats the purpose.
Hard to untangle the incentive for collusion + need to police legit bad behavior.
This could be a few large providers saying 'we control most email traffic, let's control all email traffic'. Or it could be serious players saying 'spam hurts our users, let's stop criminals using a blanket rule'.
More likely it's a schelling point where large players are rent-seeking (crowding out some competition), but only to the extent they can preserve the illusion this is about policing spam.
Suspect we'll start asking platforms to offer something like due process in the law -- administrative checks that increase the cost to administer a system, and reduce the quality to end users, but increase transparency and make it harder for the platform to engage in corruption.
"Newsletters from my alumni organization go to spam. Medical appointments from my doctor who has a self-hosted server with a patient intranet go to spam. Important withdrawal alerts from my bank go to spam. Purchase receipts from e-commerces go to spam. Email notifications to users of my company's SaaS go to spam."
Are you talking here about incoming emails? I expected that these would be reliably delivered to you and that the problem is only with emails you send to the large providers?
Part of the problem is that the wrong people are complaining.
Using Google as an example, the author has no right to push anything to a gmail inbox. Google has no contract with the author to accept mail from him.
What Google is doing, it's failing its customers, the people who signed on gmail to have an address where other people could send data to.
And now those people are not receiving everything they could, but it's only up to them to decide whether this is actually a problem and whether it's serious enough to contact gmail support.
I do understand the point and the spirit of the author, but he is actually conflating the freedom of speech with the right to be listened to.
> Over time I realized that residential IP blocks were banned on most servers.
> You just cannot create another first-class node of this network.
> Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality.
It's unfortunately true. However, the reason that how we end up like this is more nuanced than just the big players trying to power grab (perhaps) but rather because of the rise of spam/scams/phishing/malware. All big players like Google (Gmail), Microsoft(Outlook/Live.com/Hotmail), Yahoo!, Apple (iCloud) are suffering from those threats, wasted bandwidth and compute on spam detection heuristic AI.
There are industry consortiums like Spamhause and commercial entities like Barracuda to maintain blacklist/whitelist to restrict access of major MTA network interconnect to fight off spams/malwares/phishing/malware delivery from botnets and individuals. And it helps, at the mean time, it consolidated the control of who can send outbound emails.
We are seeing this trend repeatedly in other communication channels like phone calls (due to robocalls, VoIP numbers are being blacklisted by all major players' services) or Text messaging (due to spam texts, major U.S. wireless carriers band together established Campaign Registry to control who can mass send outbound text messages. This is also known as 10DLC registration).
I think the vulnerabilities of previous communication protocols (email, VoIP, SMS/MMS) lie in the fact those protocols are designed with security in mind. Modern community protocols like Push Notification has been designed with security in mind, which make it less susceptible to abuse and spamming. That's probably the way go forward.
> So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.
A MX records don't have to point to an IP; it can point to a host name.
My MX record is a dynamic DNS host name.
> Big email servers permanently blacklist whole IP blocks and delete their emails without processing or without notice. Some of those blacklists are public, some are none.
OK, but if you're having trouble sending, that's no reason to do anything with your MX record, which is for receive only. Just route outbound SMTP through someone forwarding service.
I've run my mail domain for twelve years. In that time, I've not sent SMTP directly to anyone; always through the SMTP forwarding host run by my ISP.
Well, you know, the mail is going through that ISP anyway! If I could directly connect to port 25 of various hosts around the net, I would still be routing through that ISP's hardware. So the fact that mail is routed at a higher semantic level through their SMTP server, rather than just at the IP level, just almost just a footnote.
According to Hoyle, MX records must point to a domain name. The cannot contain addresses or point to aliases. Some mailers can tolerate IP address literals in MX records, but that is not universal and will cause problems.
Unless you're doing something special, there is a big difference between sending your mail to the recipient's smtp server and relaying it through you ISP's smtp server. The difference is that if you send it direct the ISP can't read it, because it's encrypted. If you relay it, the ISP can read your mail, and even tamper with it, unless the message itself has been encrypted with something like pgp.
I believe most SMTP servers talk SSL/TLS, so, yes, sending mail directly to an SMTP server that doesn't belong to the ISP prevent the ISP from reading the mail.
Well sure, but if you're worried about traffic analysis you've got bigger problems. The difference here is that if you relay through your ISP they can read the contents of your messages. If you don't, they can't.
But this is the fault of the party you'd like to connect to; they are the ones refusing your direct SMTP connection because your IP address smells funny so that you have to resort to forwarding.
Maybe it's possible to work things out with particular such destinations. If there are certain parties to whom you'd like to send mail more securely, maybe they can give you an account on their SMTP server.
To everywhere else you go through your SMTP provider.
I think the author didn't want to say something technical about this and just wanted to say, that with changing his MX record, that his email server has changed.
My little e-mail Server on an OVH VPS is happily sending and receiving e-Mails to/from the big ones without problems for my 20+ domains. Just a basic postfix/dovecot setup with letsencrypt certificates and SPF/DKIM/DMARC working the way it should. I described everything in a short blog series at https://jan.wildeboer.net/2022/08/Email-0-The-Journey-2022/ in case you are interested.
I started hosting my own email in 2004 before finally giving up and migrating my email to Fastmail last year.
Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.
I started self hosting in 2017. (With a fresh domain. I'm the only user.) My sever closes the incoming connection if it doesn't has a rDNS or the HELO doesn't match the rDNS. Apart from that I have no anti spam measures! Friends and family have my firstname.lastname@domain address. Websites and companies get a <random>@domain address. So far I have only reserved 3 spam mails. Two times they got the address from a public mailing list I participated. One time it was from a webshop that sold or leaked my address.
I’ve noticed since switching to Fastmail from gmail my spam filtering is very problematic. I always end up with legit emails in spam, and I just can change if I want even more legit in spam or more spam in inbox, which still gets through.
I think it’s both a tough problem, as well as something that only becomes easier when you have both talent and scale to have a wide perspective.
Yeah, I ran my own e-mail server from 1995 until about 10 years ago. It used to be fairly simple, but between incoming spam and attacks, the various standards that were developed, and my e-mail not getting through... this became a problem that was worth paying someone a few dollars a month to solve for me.
I still use my own domain, but I'll let actual delivery and security experts deal with the day-to-day running of things, while I run my business (which definitely isn't that).
It was a bit sad to give up, but the time and frustration it saved have been more than worthwhile.
For my personal mail server I ended up using the free tier of SMTP2GO as outbound smtp relay. You can just register your domain as a whole, and the setup was pretty painless. They let you send up to 1000 mails/month for free, which is plenty for my own personal use, but if you need any more than that, these relay services can get quite expensive. You usually end up paying the same price as the fully hosted solution that most of those providers offer alongside the relay service.
Proper, artisanal self-hosting of email can still be viable depending on your expectations and tolerance levels for random issues.
These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.
Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.
The cloud services providers have some of the worst IP blocks for spam. Cheap hosting blocks are a distant second place.
Google for example, doesn't even hide the fact you have to request whitelisting using their online business services portal. They don't give a toss what kind of sender authentication/signing hoops you have jumped through already, and user letters may still end up in the spam bin.
Many users have indeed migrated to the web platforms, and don't care about people data-mining their business communications. The real issue is so have many spammers/scams, a side-effect keeping their trade on life support by removing technological administrative barriers for the desperate... and you can't block Google/Outlook/Redmond.
How do you know? SMTP wasn't designed for reliable delivery. You don't know if your emails are being received, and even i they are, you don't know if they'll be received tomorrow.
Statistically. I communicate with a lot of people via email, and if I don't get an expected response to something of consequence I follow up - it's far more common the recipient simply missed it/didn't get around to it than it is to get "I never got it/I found it in my spam folder".
Completely agree this is anticompetitive and worthly of examination under the light of antitrust laws. Of course the defense that will be offered and one which is partially true will be one of providing safety from spam, phishing, and other evils prevalent in the email system today. Maybe texting is the new email after all...
The real problem is and always has been "reputation". Now the big boys fuck that up by definition from the start and require you to do the heavy lifting: DNS, SPF, DMARC, DKIM and so on.
I run several email domains quite happily in the UK. I know why it works and I don't resort to magical thinking. My ISP is considered a business one and my IPs are static. I've owned both my work and personal ranges for a while.
Feel free to contact: furtle@blueloop.net - I'd love to hear your ideas.
I've been self-hosting since 2004. I currently route via a VPS. The only issues I have seem to be with outlook.com/hotmail.com/etc - free Microsoft accounts. That goes to junk, though replies seem to work fine. Paid Outlook365 seems fine.
Even after speaking with Microsoft's email admin team on the phone a couple times, I still have issues. It's kind of infuriating.
I have properly configured SPF+DKIM (selector rotated daily)+DMARC, and I've gotten set up with dnswl.org.
I hosted my mailman stack on VPS for some time, it worked well
I stopped self-hosting because it's too much hassle, but it was any difficult to maintain, (by difficult I mean complex)
It didn't worth the time I spent though, so I quit, but I would do it again if I need to
If I had to maintain a server at home and my ISP blocks it, I would get a VPS and host proxies on the VPS and use VPN tunnel to keep the mails stored locally
But I don't have any reason to do that currently, as well as most people
I already wrote this also in another thread. Which was not also only was suggested by me:
* implement and publish policies which emails you accept (regex/strings on domains, emails, headers, signatures and so on)
* found a association where all use this strict (which shall potentially be stricter than gmail and so on) settings and if gmail does not accept these emails, sue them for discrimination.
Could we come up with a new protocol (possibly based on SMTP/IMAP/whatever), that would only guarantee to get your email to its recipient if you included some sort of token generated by the recipient and given to you? Something where you could text/message/whatever a unique token to a friend/business/etc. and then they can send you email? And if you email someone, your outgoing email includes the token necessary for them to reply? The contents (including who it’s being sent to) would be encrypted by default rather than being plain text that anyone in between sender and recipient (or at least sender and recipient servers) could read. Is something like that possible?
Obviously at first nobody would have it implemented, so you’d have to get developers interested in writing server and client software, and convince people and companies to use it instead of or in addition to regular email. But I wonder how many people would be interested in such a system and whether it would be workable?
There is no commercial interest in setting up a new decentralized system, because there is no money in it. And you’d have to get buy-in from all major tech companies for something like that to even remotely have a chance to replace email. Everything else already exists in the vein of Slack, Matrix, iMessage, Line, WeChat, and the like.
I gave up self-hosting ages ago. For a while I even used the old Google for Families grandfathred in setup. That also went the way of the dodo a few years ago for me.
I signed up for ImprovMX, setup my domains there, and just route my emails to whatever service I want. I use a random gmail account that I use for my login and Google services, but the email itself is never exposed anywhere, I only give out the custom domain one.
ImproveMX handles routing for my whole family. My mom uses Outlook, and it routes there for her. If google, microsoft, or whatever give me trouble or ban me, I just quickly switch the email and nothing lost.
If you pay for their service, since they have a super generous free tier, you also get SMTP servers to use as outbound, which lets me send emails through them and not have the `on behalf of` email thing. Also they do all the work to make sure their IPs aren't blocked and in good standing with MS, Google, etc.
Another very happy (paying) customer of ImprovMX here. I have a dozen or so domains for family and friends configured this way, using ImprovMX for both inbound MX and outbound SMTP (using Gmail as the 'front end' mailbox, but using the custom domain/addresses everywhere).
I too gave up self hosting this all a few years ago - especially when hosting domains for others (family) who aren't tech savvy, I got sick of having to troubleshoot why their emails weren't being delivered. Outsourcing the delivery component to ImprovMX and the mail storage and even inbound spam filtering to Gmail, made things so much easier, even if it does mean relying on a centralised party like ImprovMX.
Well I'm slowly seeing competitors a alternatives pop up, so if it falls apart, we might have places to go. Even Cloudflare now offers a similar service. Minus the SMTP part.
I know a lot of older guys will disagree with me here, the guys that use mailing lists to work on FOSS projects, many of whom I respect very much, but I think email sucks. I only use it for the same reason I have a phone number: because some people need me to have one in order to contact me.
I don't like email. I think the problems with it are shortcomings of the protocol. I'd rather not use it. But I do, as a last resort contact method.
For me, people that use email are like people that primarily communicate over SMS. If I need to talk to you and that's all you'll use, I can. But if there's another way to talk to you I'd rather use that. Xmpp, matrix, signal, shit even telegram and discord if I have to, are preferable to SMS or email. But otherwise, yeah I have some email addresses and a phone number if you insist on doing things that way.
Author do not mention the frictions of going the other side (gmail/hotmail/live):
- you might not receive all the emails people sent to you, for the reasons mentionned in this very article
- you will receive much more spam/unwanted email. I quit using a gmail account because I kept reveiving newsletter and notifications from other people who kept mistaking his account with mine. There was probably a one letter difference in their real email address and mine. I surrendered and gave up trying to tell his relative they weren't reaching the right person, only had fun once by powning his NAS with a "cloud function", and gave up after receiving tons of newsletters and other shit.
- you can lose your account any day, for any reason without any possibility to get it back. I've seen it happen to 2 people with hotmail.
Of course it all started with spam. But the response was just as destructive. I worked for a small ISP (25K email customers) back in the mid-2000s, and it was so hard to keep email flowing. All it took was one random customer somewhere on our DSL who had a misconfigured open relay, and spamhaus would blacklist us. They wouldn't respond when we'd try to get the block lifted. We tried and tried to work with them to streamline the process, nobody wanted spam to go through our servers, but they weren't really interested in any cooperation. They were perfectly happy to stop all legitimate email if it stopped a single spam.
I don't work there any more, but I'd be surprised if that little ISP hosts their own email servers nowadays. It's so expensive to deal with such issues, it's just not worth it.
How do people that work for Google on HN continue to work there without their conscience bothering them? These are terrible monopoly abuses and you’re contributing.
It’s time for the US mail service to digitise and fulfill its constitutional obligation to provide a private, secure mail service for all American citizens.
1. Any email that isn't signed with DKIM is blocked (having a signature specific to the source email address would be better, but that is probably too much to ask for)
2. If the sender isn't in your contact list, block it, or at least mark it as spam.
3. Have an easy way to add new entries to you contact list, maybe a new url scheme similar to mailto:? So that it reduces the friction of say getting a confirmation email when signing up for something, or making sure you get emails from a new acquaintance. It would probably be good to have a way to add a full domain to the allowlist as well.
But that would make selling/sharing email addresses a lot less valuable, so there might be some resistance to that from marketing and adtech.
> Blacklists should not include whole IP blocks. I am not responsible for what my IP neighbor is doing with their server.
This is obviously laughably naive and creates infinite sources of spam.
Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.
You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.
I have self hosted for 20 years. If you want perfect delivery then completely self-hosting isn't for you. It is a good learning experience and once setup it does not require much attention.
A compromise solution is to outsource delivery which is by far the shittiest part of self hosting because of bad business practices and lack of regulation. It is the least interesting bit anyway.
My email server is setup with policies commented out to send outbound emails through another host I maintain if required. When a very large company hosting lots of email for many domains mass ip blacklisted my hosting provider late last year I used this to maintain connectivity while the companies sorted out their dispute.
It's strange that in his list of possible solutions to keep e-mail spam free, there is zero mentioning of hashcash [1].
No, that is not a shitty crypto coin. It's just a computational proof that your computer spend some seconds on hashing, which is fine if you send 1-to-1 emails like real people do, but not if you are a spammer who bought a file with 6 million leaked e-mail addresses.
See Back's 2002 paper "Hashcash - A Denial of Service Counter-Measure".[2]
It should be possible for everyone to participate in internet systems equally as peers.
Somehow it seems like the Overton window has shifted such that people find it acceptable that ordinary individuals can no longer take part in the email infrastructure as equal peers.
This is pretty much exactly the reason I stopped running my own mail. I had the mail host on a static IP, part of a block of "pristine" IPs from a local colocation operation (iNOC in Albany, NY). Ran fine for years, but I started having to call more and more customers about "why haven't you answered my email from two weeks ago?" only to find out I ended up in the spam bin. Wasn't worth it when the risk was losing business.
I moved to Proton Mail as I like their simple interface and support their goals. Pretty good service so far, worth paying for, but I do sort of miss running my own services.
If you run your own mail server in 2022, you are the resistance.
I host my mail for around 12 years so only half as long as the author of the article, I faced the same problem- that my emails land in spam for some people. In most cases it's those people who care to receive my email and I always tell them it's their provider who is at wrong and that they should switch. We have some laugh together, exchange some jokes and continue with our lives. I'm one of not many who love postfix and dovecot enough to use it to self host, but I'm fine with that, I won't throw towel and will continue to run my email, hopefully I'll be lucky enough to run it for next 12 years :) Peace everyone :)
>I implemented all the acronyms, secured antispam measures, verified my domain, made sure my server is neither breached nor used to relay actual spam, added new servers with supposedly clean IPs from reputable providers, tried all the silver bullets recommended by Hacker News, used kafkaesque request forms to prove legitimity, contacted the admins of some blacklists.
I cloned a repo, edited two lines in a yaml file, ran docker-compose, logged into a web ui, added my domain, added a couple of dns records (MX, spf, dkim, dmarc) and everything worked (yes, I can deliver emails to gmail and outlook).
I honestly have no idea why so many people say that self hosting emails is hard.
You can deliver email to gmail and outlook until you ... can't. Whether the IP block your mail server is on gets blacklisted, some heuristic shifts against you (domain name becomes "bad" and shifts a point score over a threshold for being spam), or some other external factor happens, your perfectly configured mail server will suddenly and possibly with no warning or sign that it's failed, fail.
For even personal mail this is pretty annoying, but if you're relying on mail for business reasons, this is completely unacceptable. You need to be able to assume that mail you send reaches your clients/customers. The chance of your private mail server getting banned might be low, but it's not low enough, and over time that chance only increases (especially if you're hosting from a server on big shared IP blocks with naughty tenets like on most major cloud providers).
IP addresses and whole address blocks may have a bad karma: if somebody ever sent spam from them, they may be marked as untrusted in various databases, and unblocking them is pretty hard. The filters prefer to err on the side of mis-marking a few legitimate self-hosted emails, instead of passing a spam salvo.
Maybe it's a joke, but a problem I see in every "open source self hosted alternative" is that people tend to underestimate how much work is to self-host everything.
It's either paid hosting like AWS, some intermediate docker-compose solution or your own personal server machine. In every case someone has to do the gritty work. It's either a paid service, a volunteering open-source contributor, or you.
I'll keep going with my private server (on a vps). I just moved it to another hoster, meaning new IP block. This caused blocking issues:
Some but not all gmail address delivered to the spam folder. Another big mail provider asked me to set up a web page with contact info on the same domain I use for mail. And another self hosted mail server from a public service agency had me on block list.
This caused me some headaches and I was thinking this could be the end and I have to use one of the big players. But I did not give up, invested time and it works now again!
I've been debating whether to do this myself (throw in the towel on mail). I am getting tired of fighting all the assorted battles.
This said, my concern is that the big players seem like they could, at a whim, drop you as a customer, with no recourse. This is what is giving me pause going to the big providers.
I've been looking at mail distributions like Mail-in-a-box, and modoba as an intermediate, though none of them seem to be great. Basically I don't want to stitch together several different opinionated tools into a working mail system anymore.
I was skeptical about running a mailserver but a friend set up a few mailboxes with Modoboa and so far it's going better than expected. (Mostly we just needed a mail relay.)
I did training provided by a large email security firm, and one thing the presenter said was along the lines of "this spam filter defaults to block the senders domain & IP, you can set an expiration on that block butI don't see a reason why you would". One misconfigured server sending out a single email and I assume by extension someone impersonating your domain could get you perma-blocked from sending emails to that company, and I assume it'd reduce your trust rating for other orgs using that provider.
to me, the root cause of this is just money. a LOT of people have zero scruples when it comes to money, so they consider sending email to every email address they can find, in order to hopefully make a sale, to be a perfectly valid tradeoff.
these people deny that they are causing a problem, or that they ever caused a problem, because admitting that would mean they are a bad person, and they're not a bad person! they're "just trying to feed [their] kids, man."
making decisions based on money alone is always a bad idea. ALWAYS. I do not care if it is one person and one decision, or if it is a business making a decision on behalf of their stock holders, or anything else.
if money alone is your decision-making criteria, you are making a bad decision, or you are making a decision on bad criteria.
someone always pays for everyone's scramble for money. someone always pays, and it is always an unjust payment.
in this case, spammers have cost us our ability to self-host email, which is a very significant problem, as described by the author of the linked article, with rather severe consequences if you hope to have any freedom on the internet.
so, if you work for a company that will, over time, do just about anything to get people to click on ads, you are slowly destroying the internet as it was intended to be, and was, for 2-4 decades, depending on how you define that.
Funny thing happened to me today: Gmail sent its own Google Fi customer support email to spam. Haha, wish I noticed that before spending my morning going in circles with chat support.
gmail also puts mails from mailing lists into spam, despite repeated "not spam" tagging. Wondering whether mail getting flagged as spam is even a problem anymore as people get used to erratic results, at which point we can get rid of spam filtering, or what's left of it, altogether when on balance it does more harm than good, such as preventing SMTP self-hosting.
And yet I continue to get "Walmart Confirmation Receipt" or "Verizon Confirmation Receipt" all coming from addresses like "verizon_info_nlAT2Q7uf0d@zfgfdyyqsckxbvwg.linenight.com" which means google are't even trying for some.
I see this same behaviour. It's infuriating. At this point, I'd almost rather be able to disable all their spam filtering and simply run an App Script, or email client with its own rules that then syncs changes back to the server.
I self host my private email server since 2005. Never had big problems. One time my server was wrong configured and then marked on spamhaus. Just fill in a form and all works fine again. Maybee gmail will blacklist me in some weeks, but fuck you gmail, I have nearby no one that uses a gmail account.
I had in all this year no problem with any other peer. So, maybee the writer has just a malconfigured mailserver or don't know that he is marked on spamhaus because any reason.
The range IP bans hit home. Been hit by this more than a few times. You don’t have to be sending spam, but if someone in the range did, ever, you will be blacklisted all the same, and treated like a spammer.
The people who run these blacklists are unreasonable. I can understand why, they tend to interact with the bowels of the internet and the heuristic is effective. Usually people who need to talk to them are doing something naughty, so why bother taking a chance?
Guilty until proven innocent would be an improvement.
I know of some small servers that get a lot of spam and hacking attempts, and their most effective tool against abuse is an IPv4 block ban. Increasingly this became more and more difficult, and I assume email servers are at the same point. Thanks to VPNs, people appear to be able to spawn insane numbers of random IPs.
One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.
If you have been sending credible emails with multiple hosts for 10 years, even if you did get flagged, you would be given the benefit of the doubt. Hell, it should be easy to email the host and give them the headers and the reason why the email was flagged.
About the email space now being owned by big tech, it could simply be time for a boycott until they improve their practices. There is far too much centralization on the web now, and we all contribute to it every time we use an external service rather than host our own.
> One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.
That's essentially DKIM being fed into your average domain reputation system.
> That's essentially DKIM being fed into your average domain reputation system.
Yeah, and they are not particularly happy with this solution either. A lot of people complaining, etc, as we see here. I'm still not sure of a better system.
I am still hosting my emails via docker mailserver[1]
I got some trouble with outlook.com bans, but Linode helped me to switch to a “good” ip address and it is working fine for now.
I will try ti resist as much as possible, because email is your primary identity “link” on the Internet, and you deserves to own it if you want.
I'm also using this solution very efficiently myself! So far I haven't had many issues with receiving or sending e-mails and what's especially nice is that I can easily set up as many accounts as I need, for example, one for Zabbix, one for Drone CI, one for OpenProject etc.
Even a cheap VPS is enough to run this setup (though you might need to selectively disable certain components) and the administration could be web based, but frankly the scripts that are present are simple enough and there's nothing unusual about, say, connecting Thunderbird to it.
1999 was a pretty late adopter to the self-hosting email cause since it was pretty apparent since 1997 that free email was going to win out. As Gmail filters get smarter then it becomes an altogether more fruitless cause. Email hosting at the individual level died out for most due to this and the pointless rivers of spam you had to deal with. If you've only now come to the realization that you can't make it then you've been wasting a lot of your life.
I've been hosting my email on my own server since the 90s, but got tired of dealing with keeping up with spam filters. Updated the MX to deliver inbound to mailroute.net and have them do the filtering before forwarding to my server and that's been working great for years. Not free, but not expensive and still gives me 99% of the control i want with very good spam filtering too.
Outbound mail is relayed via mailroute too, which solves the tainted IP delivery problem.
In fairness, things like postfix usually ship with very poor (not to say “moronic”) defaults.
Like, postfix won’t even try to connect to tls-enabled smtp for outgoing email by default, and you have to explicitly point it at the certificate bundle it’s supposed to consider valid.
And you have to tell explicitly to reject incoming plaintext connections from the public internet.
And quite a bit more… Like, why doesn’t postfix have its own freaking spf/dkim implementation BUILT IN?
Mandatory TLS encryption: announce STARTTLS support to remote SMTP clients, and require that clients use TLS encryption. According to RFC 2487 this MUST NOT be applied in case of a publicly-referenced SMTP server. Instead, this option should be used only on dedicated servers.
"
Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it? The only one I personally know of never does that. That one also does not use anything like an IP address blacklist. This article doesn't name names, it just waves its hands and throws around some innuendo. But as far as my own personal experience goes, this author has no idea what they are talking about.
> Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it?
Agreed, that is really broken behavior. Once you accept mail for delivery, it should be treated as a contract to deliver that mail. Rejects during SMTP conversation are fine as they notify the sender and do not generate backscatter.
I have heard complaints that Microsoft's hosted mail offerings accept then silently delete mail. It is somewhat believable, as MS Exchange server would respond 2xx for any message to any (including non-existent) destination address and later spam the possibly spoofed sender with bounce messages. Maybe MS has broken behavior like this in their hosted offerings too? And, rather than fix their software, they silently delete mail since they have finally learned that backscatter is a bad thing?
And, MS o365 allows 'delete' as an option for the centralized spam rules maintained by the admin. These mails are accepted 2xx then silently deleted.
MS does other questionable things on their 'free' hosted offerings to mitigate their abysmal spam filtering. I have a couple burner @outlook.com addresses, and they no longer receive any mail reliably from any sender. MS provides the user a place to whitelist senders and domains, but after wasting a bunch of time whitelisting domains, mail still is marked spam. "Junk" is effectively the inbox on those accounts.
Disclaimer: experience is dated, from a past job, running Postfix MTAs for a large organization and dealing with / mitigating MS issues, but never directly involved with any MS stuff.
Sending emails with dkim + sfp and I never had big problems with reachability and a postfix server on Digital Ocean.
I haven't done it since last year though.
Has something gone terribly wrong?
I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.
I managed 3+Mail at 3Com 35 years ago, and in fact, it gets its own subplot in The Big Bucks (https://www.albertcory.io/the-big-bucks), back when email was brand-new (well, for most people).
However, nowadays I'm bored with stuff like this. PITA. So I totally sympathize with the author.
My company exists to solve this problem at scale for the web hosting industry. It’s too bad that self-hosting isn’t viable because of IP reputation problems, but it’s a reality that is unlikely to change any time soon.
I’d say if you want to continue self-hosting, just let go of the delivery part. Use a service like SendGrid; it probably won’t cost you anything and it’s easy to set up.
I've been trying for the past week or so to break into the self-hosted email game since I had a VPS and found some tutorials and documentation for postfix + dovecot. I did eventually get inbound and outbound email working although I was using an alias when I think I needed to use a virtual mailbox (to both receive and send email from a user with a different name from the unix user behind it, as right now outbound still leaks my unix username).
But then I looked at Protonmail's cost and it's less than I'm paying for my VPS (which is already cheap for the use I get out of it) so I'm on the fence whether I keep hammering away at that (and then have to wrestle with the big players treating me like a spammer and do my own spam filtering) vs just pay for that convenience. The VPS is staying in any case so it's just a question of whether I pay a little more every month for convenient secure mail.
G'Day Carlos
We started hosting own SMTP servers on IBM OS2 using a BBS. Long before retail NET was in Oz.
Fast forward to today and we have a DB with millions of IPV4 address and have never used IPV6, cloudflare or a 3rd party to 100% stop spam like we do day in day out.
You are absolutely right, there are a lot of criminals you are wearing suits who are selling snake oil.
We have X simple rules, one is that inbound port 25 sessions from spammers are put to sleep and we give them a very poor service. Our servers seem overworked and they often go away never to return.
I read your blog post and from approx. 30 years experience including SMTP at gov.au and gov.nz content filtering does not work.
Our NoSpamAccepted ( NSA ) AI powered SMTP servers use the network to stop spam.
How many domains & sub-domains are you self hosting ?
TIA
KG
BTW - We have never been in the BLs as we only relay our outbound
I have been self hosting email for just over 23 years and I am more emboldened than ever to keep doing so. Even with SPF/DKIM/DMARC setup, I am constantly asking people to check their spam folder, add me to their contacts, etc. I refuse to pay an email tax to one of the larger players to solidify their hold over the protocols.
Are the big tech email servers truly not used for spam at all? Because if they are, shouldn't they be permabanning each other as hard as they ban the little guy?
Clearly they're using different rules against each other than against small email servers, and I think that's all the evidence you need to get the EU to take action here.
They actually do end up banning each other's endpoints (specific IP addresses, not entire ranges) from time to time. --Even the big companies aren't able to prevent their services from being used for spam even though they shut it down as quickly as they can. And usually the blocks are due to the blacklists like SpamHaus, rather than the email providers explicitly blocking competitors' endpoints.
Did you properly set up DKIm, Dmarc and SPF? I have been hosting my own email through a VM in Texas for about two decades. Whenever I had problems with rejected deliveries, they were solved by moving to the new standards the internet invented to fight spam. Did you use a dmarc analyzer to verify you were set up correctly?
I self-host email for a bit less than 20 years and still not given up. Being blackholed by Microsoft or Google is a real risk (though I was lucky so far), but at least I fully control receiving side - if I expect a message I can check my logs to see if there was a delivery attempt if it was not successful why.
I tried to host a home email server a few times and it was a pain to say the least.
Finally I created a droplet on Digital Ocean and used Mail in a Box https://mailinabox.email/ with a glue record to act as a name server.
So far so good
I like a lot of what’s being discussed here. One thing to consider is a similar problem I am facing is people trying to hack into a login page. We see thousands of requests per minute from different IPs… many from VPS, some from obviously hacked TVs/devices. We could implement an exponential back off on abusive IPs but detection requires observation of action that action could result in a compromised account… so another idea is we simply block large ranges of know bad IPs from blacklists… I think this is similar to the email sending issue… it’s not fair and I think a solution could be some kind of “block chain” - make it expensive to login… make it expensive to send an email… but I’m not sure and for email it’s way harder because you need agreement from the oligopoly of email providers… not sure what the solution is
There is even a much larger problem here other than not being able to self-host your own email. The issue is that large providers and large companies have no accountability. No way to ever have a conversation with an actual person (email or by phone) to resolve a simple problem or mistake in an algorithm or automated process (or even a manual process if that's what it is). Canned replies not reply to follow up question not a care in the world or a conscience. And the fact that the service is free (to someone) does not mean a company should be able to so easily do what they want and cause aggravation to others.
And it happens even with paid services at large companies.
That is not 'just business' and has never been the way business operated pre-internet except in a few super rare (and perhaps rare monopoly) situations.
The Helm email server — funded by future YC CEO Garry Tan’s VC firm, but I bought one before he returned – is a really great compromise between privacy and convenience.
The IP block is managed by the Helm co., they tunnel connections and sell you the (tiny, silent) server and software. Each Helm server generates its own TLS cert, so the tunneling does not violate your privacy (unless it was delivered without TLS, in which case your privacy already vanished upstream).
The only delivery issues I hit are sometimes with Outlook/Microsoft managed domains. It’s been at least a year since I had that issue. When I first bought one someone on gmail had to move a message of
mine out of spam, but it’s been fine since. Last I checked their infra is hosted on AWS but apparently they have some screening technique for getting clean IPs.
The general rule of thumb for my home server is that messages to people I've contacted before, or to addressees that were in a thread that I'm also part of get delivered reliably. Messages to new people I've never emailed before often go to spam. I've learned to accept this.
If, for whatever reason, Gmail doesn't like your setup, even if it's 100% according to specs, it's effectively broken. And there's no one to resort to, often no previous relevant search result, because the errors are vague when not silent (you don't get to know the email was hard rejected, i.e. has not even reached the recipient's Spam box).
Your only hope is for your complain to go viral on HN or Twitter and some Googler takes pitty on you.
Not to mention that your email is tied to your online identity in many ways when mostly every account you make asks for an email to use as your username, verify your account, or recover access to your lost account.
Handing over control of your online identity like that to a centralized third party when you could be cut off at any point [insert here any recent news about people wrongfully losing access to their Gmail account and not getting it reinstated] seems like the wrong solution to be allowed the privilege of sending email.
I guess the very least we could do to keep some control is to own (read: rent) your email domain you could move elsewhere in case you lose access, but then you gotta make sure you don't also lose access to your email domain.
Giving large providers a taste of their own medicine might be interesting, let gmail spend 24 hours on a black hole list, make sure the removal forms have a captcha so they can’t be filled out by automation, when people complain on social media then let people know what it’s about. Social media really gets Google’s attention, it draws them out from behind their algorithms and silence when nothing else will.
Utilizing IPv6 more might help also since at some point it becomes absurd to have quintillions of addresses blocked for all eternity.
There are monopoly and racketeering angles also, since Google and Outlook are suppressing independent mailers in favor of their own paid products. Nobody will make any money except the lawyers but that’s ok as long as the situation improves.
Racketeering is when the only solution offered to you out of a problem someone created for you is to pay the person who created the problem. In the context of email: "you have to pay us to send your email because we have created a system in which only we can send email." Seems pretty close to racketeering.
Anyway, telling someone they don't understand what something means is almost never a good way to start a conversation.
That’s not what racketeering is. In order to be racketeering it must be extortative, fraudulent, or otherwise illegal. Your definition would make PG&E guilty of racketeering, which they are not. Selling fire insurance is also not racketeering.
When someone is using a word incorrectly making them aware of that fact is important. They’re welcome to google it.
How would selling fire insurance or an electric hookup match my definition? For fire insurance to be racketeering the ones selling the insurance have to be the ones starting the fires. For PG&E... IDK they'd have to have invented work (J/s)? From the Wikipedia:
> Originally and often still specifically, racketeering may refer to an organized criminal act in which the perpetrators offer a service that will not be put into effect, offer a service to solve a nonexistent problem, or offer a service that solves a problem that would not exist without the racket.
It's unlikely (IMO) that GGP meant that they thought it would literally meet the legal definition of racketeering in a court room, more that the situation with email can be described in terms of the textbook definition.
A big problem with the way the big companies fight spam is that sometimes it is a bit too aggressive. Happened a couple of times where legitimate government-agency emails had trouble getting through.
Personally, I prefer defining my own spam list rather having an algo decide what pops into my inbox.
What do I do? I got rid of Spamassassin after being challenged by someone on HN or Reddit, can't recall which. Spamassassin is pretty effective but was consuming too much of my time.
So I stopped. I analyzed mail for a few weeks to look for patterns in the wild with my server. I came to the conclusion to block all but the top level TLD's. That decision yielded very positive results.
I then wrote a simple SPAM blocking server to allow me to block habitual or suspicious TLD SPAM sending domains, as well as a few custom checks for common sense things.
As a result of those two decisions, I am now at or better at blocking SPAM than I was with Spamassassin and 2 other related tools I just remembered I also stopped using, spamass-milter and postgrey.
RBLs and other blacklists have been annoying for decades, even more so when they were maintained by annoyed individuals rather than corporations like today. You had to beg a single person to remove you off a list that an ISP somewhere used without thinking twice about it.
The name evades me, but this was a thing a few years back. IIRC, it was the equivalent of what a stamp for an envelope would cost, and the recipient got a chunk, as did the service provider. Will try and find the name for you, but AFAIK, it's no longer doing business.
Alas not! Much more recent, had a very polished of-its-era SaaS style website. IIRC, you might have even been given a specific email address for it, which then forwarded to your chosen personal address.
>In many countries politicians are forced to deploy their own email servers for security and confidentiality reasons. We only need one politician's emails not delivered due to poorly implemented or arbitrary hellbans and this will be a hot button issue.
"I just the other day got... an Internet was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday!", as Ted Stevens would say.
Unfortunately the man said this as part of a massive, uninformed speech[0] about why big tech[1] needs less regulation.
I'd like to consider migrating from my self-hosted solution but it seems like all major email providers want $ per month per user|mailbox. Are there any providers out there that charge a reasonable fixed amount for say 50 mailboxes?
OVH MX plan? The so-called "pro" mail is just outsourced to Microsoft though. 50 mailboxes is an odd spot, too much for personal providers and too little for others.
Same here, I’ve been hosting my email since 2007 and gave up in the beginning of this year, mostly due to constant delivery issues with microsoft infrastructure, which can be mitigated for a couple of months before coming back again
> The industry should fix email interoperability before politicians do. We will all win.
Not sure if by "politicians" he means legislators, but given very few players that control today's email deliverability, while doing very little to provide observability (=feeback loop) to the users who needs it most (that is users who cannot afford to build an expensive pipeline that optimize deliverability), given all that, I think regulation around distributed protocols observability/fairness is not unlike AI explainability regulation, only I expect that with mail it shouldn't be as hard to implement.
You start with the premise that you can't host your own email, which is problematic. I don't know why the people who fail at self-hosting email are so adamant about telling others that nobody should self-host, but it seems more like a squeaky wheel problem than a real one.
In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
You get all the benefits of your own filtering, your own logs showing every delivery attempt, you get to store your data on your own systems, access it however you prefer, et cetera - all the reasons to self-host are there except for delivery logs, and those can be arranged through your smarthost provider.
Simple, huh? So why are so many people emphatically telling us to NOT self-host email?
Sure! I would never recommend, for instance, Dreamhost, but I set up a company's smarthosting with Dreamhost because the company was already using them. Dreamhost in general, though, is quite spammy.
Linode and Panix come to mind.
Really, though, many hosting companies won't necessarily list "smart hosting" / "smarthosting" as a service. If they offer a cheap VPS and offer outgoing SMTP, you can set up smarthosting through a VPS.
One can even smarthost through Outlook / Gmail, if you really want.
Sending via a VPS is not the same as sending through the VPS provider's email servers. Providers usually have their own outgoing SMTP servers for their clients with people who make sure that deliverability is always working, so sending through those is much more likely to work than sending directly from a VPS.
You're in control of all deliveries to you, so you can grep through your logs and see any and every delivery attempt, which you can't do with, say, Gmail.
You can store your data however you like, with as many (or as few) security considerations as you please. Want full disk and OS encryption with a Yubikey that has to be physically present when you boot the system? Sure! Want to encase it in a cube of concrete? Install your server at the top of a tree? Why not? Want to store all of your email encrypted in memory? Go for it!
You can back it up as you like, you can make sure nobody else sees or indexes it, you can access it via command line, webmail, IMAP, POP, whatever. You can more (less) your spool file directly. You can overwrite the disk where the mail was stored when you delete it. It's up to you.
Smarthosting does mean you have less control at delivery time, though. For instance, I can choose to refuse to deliver email to domains that don't negotiate TLS. Smarthosting doesn't easily allow this (or at least I don't know of any way to do this).
But every other part will still be 100% in your control.
Yup. I have to pay Google a chunk of change every month to do something I used to be able to do for myself for free. Because suddenly they flagged my domain and no one would get any of my e-mails.
I've been self-hosting my mail for 17 or 18 years now, by purchasing just some managed webhosting package from somebody who cares about their services not being used for shady stuff (any reputable managed hosting provider) and I think I've never "lost" an outgoing mail for personal use.
I don't understand what the author thinks it's so hard here and why he's painting it so black and white. There's lots of more to "my own e-mail" than choosing between some old notebook running and collecting dust in your garage and using GMail.
I've been using exchange online for my personal email. I had to do a bit of fenagling with my dns records, which is old hat at this point, to get it to work but it basically costs $4 per month per user and since I'm the only user it gives me the freedom to use my own domain name and not have to worry about selfhosting. Prior to that I was using exim as my SMTP server and dovecot as my imap/pop3. But managing it was a huge hassle. $4 a month is worth the effort of not maintaining it myself.
> Email is now an oligopoly, a service gate kept by a few big companies which does not follow the principles of net neutrality.
While I understand what they're getting at here, I disagree. There are certainly other providers you can go with - You just need to pay up.
Happy customer of FastMail here. All of my personal domains, and multiples businesses are tied to it. Wonderful service, no deliverability issues, great features, etc.
So - clearly other companies are able to get it right. Self-hosting is probably difficult, but it's not like you're forced to go to Big Tech.
Yeah, this is my compromise too. I'm not able to spend the effort setting up my own mail server right now, but don't want to just use one of the really big players, so I use my own domain and FastMail. Feels good to support a smaller player.
Just did a test with my own mailserver hosted on a small-ish local vps.
Outlook: OK (had to bother with this one when I started out)
Google: OK
iCloud: OK
I have a pristine track record and not a single byte of outgoing spam so I cannot attest how easy is it to get back into the game after an incident. I do agree with the larger point being made here. It is clear some kind of anti racketeering legislation would be the only fix. Sadly, currently there is zero will from both the EU and the US to fix any of these blatant anti competitive issue on the internet.
My experience is that Outlook will accept email from a new IP ... for awhile. But rather quickly they block it. I think they watch for a commercial amount of traffic, enough to get a reputation from various services. Not enough traffic for a reputation? Very suspicious, adios... How else would legit commercial email senders bring up a new server?
From the MailOp commercial mailers mailing list, it seems Outlook will eventually unblock you, but you have to keep appealing, etc., for multiple rounds.
The dream is a self-hosting email, which is also super easy to setup, so much that any person in the world could do so without specific technical knowledge.
I just moved from fastmail to self hosted and it works with no email blocked so far. Had that setup for 6 months. The most important point being sure to have a static IP with reverse DNS.
I am running my own mail server for 3 years now. There were some issues (outlook pushing emails to spam folder) and an IP change (not sure how that affected things but I resubmitted the IP to the Microsoft email tools, can't remember the name of the service).
I would like to join some movement or a class-action lawsuit against big tech for anti-competitive practices regarding private mail servers if such thing exist, and if not i would suggest we make one.
I love the idea of self-hosting my email, but there's no way I'm going through all that work if getting my mail delivered will be a toss-up. Complete motivation-killer.
> We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.
What would they do with email?
To be honest, I kinda like seeing a lot of cookie modals out there. Yeah, the experience can be hellish, but it highlights how many sites are actually collecting data from their users.
With that said, I wonder what alternative regulations are feasible if we don’t rely on politician-mandated regulations.
I also came to the same conclusions 8 years ago after hosting my own email for more than 2 decades. You just can't do it on your own anymore. There are some proposals how to fix this, but I think a more sane approach is whole email system overhaul/replacement with something better. Many people are thinking about this and come up with good ideas. I hope eventually one of them will get enough traction to replace our frankly outdated email technology.
Part of the problem is silent deliverability issues. You can send to someone @gmail.com that you've never messaged before and it won't get rejected. It will probably end up in their Junk folder. You don't know it, they don't know it, and unless they check, which many people don't, it will be as if you never sent it.
I am somewhere in the middle. Have been running my personal mail server for 15y, it mostly works but my emails do get often flagged as spam by the major providers (but not deleted). Though those very same providers are themselves a major source of the spam I receive. Do as I say, not as I do.
The nice thing if you control your domain is to be able to create unique email aliases, which is a way to cut spam to zero. A company starts spamming or leaks your email address, just delete the alias.
At what level of the net does IP address reputation operate? Do the blocklists ban whole ASNs based on some measured volume of spam?
More transparency (or more likely, less ignorance on my part) here would be helpful.
Can anyone recommend a one stop tool / script for looking oneself up in the reputation services? (And on that note, it is abhorrent that Big Email providers don’t have open reputation databases, or at least ones where I can look myself up.)
«So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.»
I would have kept the MX records pointing to my personal server, and I would have changed only my configuration to send outgoing email through a third-party relay (eg. Gandi). This would have solved all the author's problems (deliverability issues) while staying 99% self-hosted.
BTW, did you know the smtp protocol works without DNS?
You just need to puth the ipv4 between brackets @[xxx.xxx.xxx.xxx] and for ipv6 @[ipv6:...].
spam? simplicity and freedom has a price (personnaly, I have have very, very little spam since I am self-hosted), and don't think corpos won't try to force you to use their servers one way or another... Whose coding the virus? It is sane to presume it is the seller of anti-virus software...
>BTW, did you know the smtp protocol works without DNS? //
I did but assume no-one uses that as it's not practical because most IP addresses serve more than one domain? Or does SMTP handle that, like some extra "really-to:" header to encode the user@domain if you use IPvX for delivery?
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.