I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
You don’t have to use blocklists to block messages.
I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
I don't use any kind of block list on my mail server, as I find the concept to be fundamentally flawed. I accept all incoming email as long as basic sanity on the connection is met and then apply bayesian filtering after the email has been accepted.
I get just about no spam at all (<10 per month, maybe).
Any guess on why people's experiences with spam are so diverse? I self-host and get fewer than 10 true spam emails per year (not counting marketing newsletters).
> Any guess on why people's experiences with spam are so diverse?
That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.
> > Any guess on why people's experiences with spam are so diverse?
> That would be an interesting research project.
Indeed. When you say get spam, do you mean pre or post-filtering?
Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the same time period there was only 1 spam email which was missed by spamprobe and made it into my inbox spool.
There's a fair amount of would-be spam that gets blocked during the SMTP transaction due to things like bogus host in HELO, etc. I don't keep any stats on those, I did at one time but it was too much noise.
My email isn't secret, it should be on every spammer list I'd guess. It has been the same since the mid 90s and is all over usenet, email list archives, websites, etc and I've never made any effort to mask it.
You mentioned receiving 10s of spam per day - are they being classified correctly and put in your spam folder or are you seeing them in your inboxes?
My mx hosted with hetzner also runs rspamd. Of the 32k mails received in the last month, 40% were rejected (postfix DISCARD, so the sender sees the mail as accepted but its sent to /dev/null - this only happens to mails scored very highly as spam, or sent to a spam trap address), 10% were greylisted and 1% were delivered to the spam folder.
So I'm also receiving 10s of spam per day, but they're all delivered to my spam folder with rare errors.
With the exception of an issue delivering to AT&T recently, I haven't had any outbound deliverability problems in a few years, but then again I don't send very much mail at all - perhaps I'd have more trouble if I did but most of my mail is incoming.
> are they being classified correctly and put in your spam folder
Mostly. There's probably 3-5 spams a day which get into my inbox; also 10-15 a day that wrongly end up in the "maybespam" folder (generally bulk that I'm not worried about seeing 100%.)
> That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
The old lists may be more plausible.
Some spammer once associated 'Robin Bennett' with my email addresses at some point 20 years ago and kept reselling it. Never used that name, didn't even know Bennett was a surname. I assume they linked me up with that name to fill up empty cells in their list and make it look more plausible.
It's a good way to filter spam. The last decade almost all spam calling me 'Bennett' is from some US political group, which mostly reminds me that it will be a long time before anything GDPR-like will pass in the US..
But almost all spam goes to that old 20 year old email address I don't use. My current email addresses are much cleaner even after 10+ year of use
> I thought self hosting is for aliases? Shouldn't those fix spam for good
Not quite sure I follow? Unless you mean "only allow emails to specific email addresses you've noted down", in which case, yeah, that works but also means a lot of admin when you want to use a new one (plus there's 10+ other people who use my servers for email, not just me.)
> unless you publish your address globally?
At least one of my email addresses has been published globally since ~1995. Others since ~2000.
> By lot of admin you mean that it's not automated? That's sad indeed.
How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?
Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to <randomword>.20.<knownaccount>@spamgourmet.com and you can't protect against those 20 spams.
But I don't find that burdensome at all, in fact I'd prefer that gmail let more stuff through to my spam folder instead of swallowing things that it misclassifies, because I have seen several instances of lost legitimate email, something I still don't understand how is deemed acceptable to the people who wrote it.
But I need to vote with my feet instead of complaining.
> One spam every three days is insanely high IMO (compared to what you will get on gmail)
That's funny since on gmail I get tons more spam than on any of my self-hosted domain addresses. What's worse, I see emails in gmail misfiled as spam when they are not, which is far worse. I don't ever see that happen in my self-hosted system.
And for most people, using a large mail provider is the necessary evil. I don't see much difference between a small mail provider using block lists and a large mail provider using the same block lists + their own proprietary lists and metrics.
Both result in other small mail providers getting blocked for arbitrary reasons with limited recourse.
Using large mail provider is not necessary evil in my book. It is just so much more convenient than self hosting.
Today, with ready-made containers and stuff we start having a fighting change, but the moat is possibly too big to fight with email falling out of favor
I think that gray listing is a great idea to filter out spam. Here's how it works: the very first time an email server get an email from a new IP it answers with a "temporary unavaiable" error. According to specs, a server should retry to deliver the email after a while, so a legitimate server will retry and it's IP will be put in a "withelist" (here in quotes because you can still do further processing of the emails to determine if it's good or bad). But a spammer will most likely not retry to send it, as their goal is to quickly send a big amount of mails.
I didn't try this personally as I'm not hosting my own emails (I tried but gave up soon), but I heard it works very well, no third party blocklist needed.
The trade off is that even legitimate mail that you are actively expecting (like an account confirmation or password reset mail) will be delayed by however long the sending server's retry interval is.
You also either need to apply the greylisting to some larger IP range (rspamd e.g. apparently uses /19 by default for IPv4) or otherwise specially handle some of the bigger mail providers, because some of them rotate through their servers between retries, so you could be in for a quite a long wait if you do per-individual-IP greylisting.
The biggest culprit I noticed this with was Amazon SES – a former mail provider of mine used per-individual-IP, non-configurable greylisting, and any mail sent through Amazon (which isn't just Amazon itself – quite a few companies are using Amazon SES for transactional mail and suchlike) would consequently almost always arrive several hours late (however randomly long it would take Amazon to finally re-use an IP during a subsequent retry attempt).
Even more infuriating, my mail provider's support would then claim that it wasn't their fault and they didn't know anything about any supposed greylisting.
Mostly. But then you get the "click the link in the email within 10 minutes" problem. There's also a non-zero number of "our mail didn't get through first attempt, oh well, give up" people. From running GL on my servers over a couple of years, it mildly cut down spam (on top of blocklists and fail2ban) but I'm now wavering over whether it's worth the hassle.
It cut spam tremendeously for me, but that's on an email address I've used and published openly since early 2000s.
Still, I've given up on it since plenty of email senders are not standard-abiding (they fail to retry), and I've kept losing email. I only caved in in the last 12 months after 15+ years of doing graylisting.
I have greylisting enabled for years. But recently some Russian spamers were able to circumvent those by using compliant SMTP servers. They even support proper SPF, DKIM, DMARC and all that stuff what you can think of.
That is the reason why I switched on some external block lists into the mix.
I wonder if anyone compared the effectiveness of postscreen vs greylisting. Can't find any
Both rely on filtering out non compliant senders, but postscreen's filtering might be less disruptive. Are there spammers out there who cannot pass a graylist but can pass postscreen ?
What we found 20+ years ago when we started to provide a very poor service to attempted spam delivery was that the spammers realised our servers were very 'slow' and stopped the attempts. We used to get 1000s of new IPV4 tuples / day, now we get < 10. If you put the inbound SMTP sessions to sleep this is wrecking the spammers bus. model, they need to deliver 1,000,000,000+ / day. Don't u fell sorry 4 them. The only time we don't tuck them in 4 a snooze is if they are in a 3rd party BL !!!
I don't follow. I am pointing out that using the same kinds of lists that the big providers use will result in smaller providers being arbitrarily blocked with little recourse. Do you disagree with that assessment?
First, yes. They (the small providers) have recourse, it’s just annoying. They voluntarily signed up for that annoyance because of their ideology.
But much more importantly, that question is orthogonal to my argument. Using blocklists is a good way to cut down on spam, the fact that it might block some trivial percentage of people who for ideological reasons might or might not be on those lists isn’t the receivers problem.
You want this to not be the case? Contribute meaningfully to solving the problem in a more effective way. Don’t blame the people who just don’t want the spam.
The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. That "trivial percentage of people" make up a substantial portion of service providers (for every Google there are many <100 user mail servers), which is what this thread was about.
I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
>The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. [...] I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
I don't think it's the tone. While the inability for senders to get off blocklists can be true, you're still not addressing why the cost to the sender to not be blocked should have higher priority than the cost to the receiver to avoid blocklists to make email admin more of a burden.
The gp you first replied to (PinguTS) is running a personal mailserver and resorted to using blocklists because reducing spam -- at the cost of some legit people not being able to send email to him -- is a tradeoff he's willing to make. You haven't convinced every user running mailservers that they should increase their spam burden because some small providers can't get off blocklists.
As another example in another communication channel... Here's a similar "blocklist" for cell phones that some users take advantage of: https://about.att.com/pages/cyberaware/ae/cp
Are "legitimate" phone numbers getting caught up in that block filter?!? Of course. But phone owners are trying to stop spam telemarketing calls. Telling them that some phone users are incorrectly on AT&T's list isn't going to convince cell users to quit using the block filter. They're willing to live with a few legit callers getting blocked.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
The correct response would be to set them on the blacklist for a time. People would complain and they would quickly change their behavior. Microsoft currently really wants everyone to have a Microsoft account.
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns
v=spf1 mx -all
This tells me only to accept emails from 'MX' entries for that domain. So I query 'MX' against the DNS server and I get a list of A records, which I can get IPs from. If the IP is in the list, spf passes. Otherwise it fails, mark as spam.
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
DKIM selectors aren't UUIDs. You can of course use a UUID as a selector, but you don't have to. My selectors are named S-YYYYMM (when I rotate the keys), so my current public key is at S-202001._domainkey.example.com.
A lot of tools generate UUIDs for the selector, just to get something unique without having to ask the user for something relevant or defining some other heuristic. For instance: the built-in helper tool for Zimbra generates a UUID by default, unless you provide something specific. I think a lot of people assume it should be a UUID just because they see UUIDs used in common examples.
Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
Counterpoint, our mail admins spend a lot of time trying to convince small-scale providers to shutdown the spam email coming from them. Lots of people who host at small scale providers don’t care about patches, so they send tons of spam.
Doesn't seem like a counterpoint to me. The provider you're describing isn't clueful. A clueful provider pays attention to what's happening on their network and knows how to make themselves an unattractive host for spammers.
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
> I have been hosting it on a bare metal Supermicro server in a proper datacenter, though.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
I've run my email server in VM's at a variety of ISP's like GoDaddy, Linode and AWS Lightsail and never had an issue, well once when Microsoft banned a group of IP's in a drag net. I filed a request with them and they fixed it in a week or so.
Real hardware servers are not required to host email.
I had tried it on a very inexpensive VPS before having my own physical server. The IP was blocked almost everywhere. Even though I filled out a lot of forms in Outlook, Yahoo, etc, it appeared that those major email providers had blacklisted the entire subnet of the VPS provider, and IP reputation seemed to have made little difference. Choosing the right provider is crucial!
A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.
Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.
Selecting the right host is very important. They should be unattractive as a host for spammers. Get this part right and you don't have to worry that tomorrow your neighbour is a spammer. The virtual/dedicated thing is not as important — many providers comingle virtual and dedicated servers in the same address space anyway.
One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.
I don't agree, but it is true that some aggressive DNS RBL may include Digital Ocean, for example, just because [1].
The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.
1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."
I don't think it matters which provider you choose. Big corps ban entire IP blocks, and you will get spammers as neighbors both at a low-cost provider and at a "proper datacenter".
FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.
It absolutely matters which provider you choose, assuming the provider is big enough to have their own address allocations (which is not that big, really — plenty of relatively small hosting providers all over the world have their own ASN and address allocations).
My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
You say that self-hosting a mail server is obvious and extremely low effort, which simply is not true. But ok, let's say that you happen to have a clean IP and no spam or dmarc issues. You still had to choose what to deploy, which is not obvious and actually setup/maintain your server which is not extremely low effort.
Please read the comment you are replying to again. I said that it's obvious that it's worth doing, not that the work to set it up is "obvious".
Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
I understood that sentence as something that can happen to you even if you believe it won't.
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
I agree, the core of the enterprise suite of Microsoft is still very much based on domain controllers and Exchange. This enables the needs for user management and access policy configuration. Where I live self-hosting with Exchange is still the usual way too. No serious company would ever use Gmail.
I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
I hope at some point the routine crushing of smaller providers gains the attention of the competition and markets authority, or its equivalents. It is crazy how Google and MS use their market power in this way and get away with it.
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
2. Add support to the SMTP protocol to let the receiving server demand that the sender make a small "postage" payment. This will require a privacy-preserving micro-transaction system. I worked on one that doesn't use crypto [0].
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
4. Add SMTP responses that mean "rejecting because your server sends spam" and "rejecting because that user sends spam". Servers can mark sent emails as "Receiver rejected this email" and inform the user.
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?
G'Day Femto
Over the past 15+ years I have joined a number of groups like a web site 'web hosting talk' and all of them jumped all over my privacy and passed on my details to spammers / hackers.
I know what you are thinking -> how would he know that ?
Well its really simple I use DedicatedEmailAddressing ( DEA ) and our system tells me when a 3rd party tried to deliver a spam message using one of these DEAs.
Some of our customers charge suppliers for a new replacement DEA when they find the supplier has 'leaked' :)
Yeah, the host I use sells on webhostingtalk and I'm 99% sure they're selling my info on. My cPanel username there is the only place that I have that username, and that's what a massive chunk of my spam is directed to.
I'm a newbie to YC, can't c 'Edit' button, so I'm assuming it is like Twit. I'm not saying I wouldn't be interested, I think it is a really good idea, but really concerned that I don't find it is a waste of time after my privacy was again splattered
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
> Host your own MX servers but use someone else's SMTP servers.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer.
It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads.
The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
>I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
> I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
I remember in grad school, one of my classmates being upset about getting a porn spam email in Chinese and saying, “but don’t they know that I’m a girl?” I pointed out to her that I got the same emails (but had no idea what they were saying) and explaining to her that they send them to everybody.
It can very much depend on your domain. In the early 2000s I ran a domain for a firm and the domain had 'UT' in it. Well spammers thought it was related to university of Texas at some point and we went from around 10 messages a day to over quarter of a million. The immediate issue I has was poor back scatter protection so I had to configure that in a few days. That stopped 99.99% of the spam, but still caused massive issues with the mailboxes that did exist getting hundreds of messages per day even though we were blocking tens of thousands of mails per valid email addresses.
Spammers are miserable, I don't fault anyone for giving up.
SPF and DKIM no longer have any real value. Just look at SA and the values it assigns by default to mail with those headers. Couple that will the fact that spam now has valid SPF and DKIM - just makes them pointless.
DKIM and SPF, along with DMARC are more about authentication - they let you know that the message has come from where it says it does. This makes spoofing harder, not necessarily spam. Greylisting deals with most of the "spray and prey" spammers, though it does have it's own issues.
I do this too with a different provider whose whole business is relaying for self hosters. I have had exactly 0 problems since I started a couple of years ago. Its kind of perfect.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit.
Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Adding to the happy medium is to teach your friends and family to use Thunderbird so they can easily GPG encrypt [1] their emails keeping the nosey email providers off the email body. Also teach them to use the IMAPS (TLS) endpoint for their mail provider, usually port 993. There are probably simpler how-to's with pictures, I just do not have any of them handy.
It could be we have different circles of acquaintances. I have managed to get non technical friends to GPG encrypt their emails. I also talked 2 lawyers into using this and the two lawyers are not only non technical but have nearly zero patience.
I think you’d be very surprised at how much lawyers don’t know or care about any of that. They store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents).
> store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents)
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
They do. Most of them use proprietary https web interfaces that usually have "secure email" or "secure messaging" in the description but they are just fancy web portals. I despise those systems. The content is not encrypted at rest and can be leaked. With OpenGPG the emails are only decrypted on the recipients end points and can be deleted by request.
Not all of them. I keep seeing lawyers sending photos of sensitive documents using their cellphone and Whatsapp (before encryption), and most of them don't even care about all those documents still residing on their cellphone, at the mercy of whoever steals it since all the security they have in place is that swipe thing that anyone with good eyes spots in 2 seconds straight.
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
Perhaps you might care to share with us what types of devices they do have, seeing as my mental telepathy transgizmatron happens to be in the shop presently.
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
there is definitely software to help with pgp on ANY tablet, phone, or PC your friends and family have access to. The universal problem is no one cares to encrypt except a few computer nerds, spooks, and journalists.
It means you now have to worry about key management or else lose access to all your old mail, it possibly complicates logging in on new devices or via webmail, partially breaks server-side (spam) filtering, breaks server-side full-text search… so there are some definitive trade-offs there to be made.
Not a single member of my friends or family is going to use GPG encryption, not even my mother who loves me dearly. I had a computer geek "penpal" for a while and we did monthly emails to each once a month to catch up via gpg, and even that died off lol
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
To be pedantic, the version I linked to is a paid addin. However, you are correct - there is/are unofficial unpaid and unsupported versions available that I was unaware of, per the FAQ and the repository that you linked, which has nothing to do with GPGTools GmbH; from the repo, “This does not constitute any endorsement or promotion of our releases by the respective copyright or trademark holders.” Thanks for sharing this.
> You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
I've been hosting and operating my own MTA MX with Postfix since 2005. I have Postfix set to use "MailDir" format storage (one file per email) and then use Procmail filters to direct emails into per-sender or per-topic specific folders when they arrive on the server - nothing needed to be done in the email client. Dovecot provides the IMAP4 client interface. Thunderbird connects via IMAP4.
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
No...it isn't the only sane choice. If it works for you, great. Forunately for Internet freedom, there many of us small timers left who have no issues hosting our own email.
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
Indeed. I host my domains with dreamhost and it turns out that outgoing mail from their servers will get marked as spam by Google. The exact same mail sent through a gmail address (whether under gmail.com or a custom domain) will be delivered no problem (although after the sudden closing of legacy free email, I found that emails that I had been sending via gmail with my own domain that had been getting blocked by spam filters were being delivered when they came from a gmail.com address).
The problem is your outbound mail server does not have a good reputation. You also probably do not have a large enough swath of IP space to prevent bad neighbors from becoming your problem. Lots of tricks to getting your email sent reliably to most mail servers. What got me out of hosting mail was trying to send an email to a potential lead I met at a local meetup. His email was hosted at some very small and relatively unknown university, but my email was flat out rejected. Something about that moment just clicked that it's not worth my time to chase down the admins and resolve the problem. I'll just start shunting my email through O365. I still selfhost everything else I can, but I offload mail management to a provider
It's more than that. I'm on a mailing list for a social organization that I've belonged to for 20 years, and has been hosted at the same place for all of that time. I have marked hundreds of the messages not spam, and still Google sends about 80% of them to spam.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
As a counterpoint, I host a couple of domains with Dreamhost and have had no such issues. They've been serving the email for my main domain for 15 years and AFAIK I've never had any of those emails go directly to spam within Gmail or elsewhere. I do periodic smoke testing by sending test emails to different place, plus I have a reasonable number of external users making use of the domains for email as well. FWIW I moved all my DNS records to Cloudflare, but that was only in last couple of years.
Sure there is.. unless you mean no recourse in this environment. The natural recourse would be to force Google to divest GMail from it's core business.
No, the recourse is to force net neutrality. You must process all emails the same way and they must go through the same filters. In fact if spam processing is part of modern email it should be part of the standard and then all mail servers must be forced to conform to the standard.
I just randomly looked at 8 different emails in my inbox. All of them were from different email providers (except google which was there twice). There's hundreds or thousands of email providers you can chose from.
iphmx 1
google 2
kornet 1
linkedin 1
secureserver 1
amazonses 1
self hosted university email 1
True. But they are not calling for a completely open system. I like this proposal from the author.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
> I don't mind doing some paperwork or paying a fee to prove I'm legit.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
I can just imagine the headline. “Ask HN: Google sent my mail bond to charity for no reason and has torpedoed my small business, and I can’t get in touch with anyone to make it right”
I can imagine headlines like that too, but the idea of electing 5 (or some other odd number of) entities is that they would be able to share among themselves the cryptographically signed evidence of the spam they detected, and then the bond slashing would require a majority vote.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
You're describing one of the solutions made by Ironport, "Bonded Sender". Their solutions were sold to Return Path and Cisco later bought them out, presumably with the bonded sended solution still belonging to Return Path? [1,2]
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
Is there any actually money to be made in hosting email for people? I genuinely don’t know but my suspicion is that GMail, Yahoo, Outlook, et al are loss leaders for their owner companies. I suspect people at those companies would be quite happy if the protocol got unfucked enough that it small players could participate without negatively impacting the network.
Spam has been been fought for decades, you can rest assured any obvious solution has been tried and either doesn’t have the desired effect or is impossible to implement.
You ignore the fact that there are perverse incentives among the participants. It's possible to implement, and I'm doing it myself. If I had more time to spend on it, we could end spam. Instead I am fine as is: most of the spammers have given up.
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
signup confirm emails are not what i'm describing, because you need to establish and filter the initial offer that they send you via email itself, which is still prone to phishing.
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
This is useless because users are stupid, people sending the mail are stupid, people getting the mail are stupid, UI people creating interfaces are so stupid society could be improved by putting them in a box and mailing them all to some wasteland and hoping they form their own society there or starve.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
This makes sense for service emails and is similar to how push notification services like Pushbullet work, but can't work for humans. You need to be able to give your email to someone IRL so they can send you a message. Mutual approval would be possible in the "we just met and want to exchange emails" situation, but that too breaks when you legitimately want to give anyone the chance to message you.
> but that too breaks when you legitimately want to give anyone the chance to message you.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
I know, but anything out of band won't really work, because that can be phished even more, plus as described above, there's no real need for it either (IMHO).
That's essentially how www.hey.com works! I thought it would be tedious at first, but I don't mind it, and it's done a great job of making it so I only see what I want to see in my inbox.
From a quick read, hey.com still allows arbitrary people to message you and just initially puts them in "The Screener", which doesn't seem like quite the same thing as an absolutely "completely whitelist (opt-in to receive messages) basis".
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
Spam has been a thing forever. Literally forever. Yet people have been dealing with spam far better than the big boys, and doing so for decades.
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
I'd imagine they do this to not tip off spammers when a message goes through. It's the same idea behind returning a 404 when trying to access a resource you don't have permission to, or not telling users if an account actually exists under an email when doing a password reset.
It's simplistic for a spammer to tell if mail is getting delivered to end point. EG, just have a gmail account, and spam that account. Simple. Done.
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
A normal company could engage in these practices, but it is settled law that a monopolist may not. A monopolist may not use their monopoly in one are to supress activities in another area.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
Substitute “HTTP” with “SMTP” and you already have that today… SMTP in plain text isn’t generally used anymore, it’s always transported over TLS. Of course you can run whitelist-only, but how is that ever going to work?
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
Whitelist is easier. If I added an mail of a friend in this system, we can communicate. Everything else goes blackhole.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Of course whitelist is easier, but how valuable would your email be if you only allowed your friends to email you? Goodbye order confirmations, subscription reminders, recruiter emails, notifications…
Sounds like a dream, really. And you could always have it be user-driven action, like "Copy this line and paste it into your email provider's whitelist", framing it as a positive win for the user since they have total control over who is allowed to communicate with them.
I do want arbitrary people to reach out to me regarding my home page for example, so any just-allow-whitelisted-messages proposals are useless in that regard.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
Certs for MX servers are supposed to have the MX as subject or SAN, not your email domain. It's important when the sender enforces encryption with a valid cert (e.g. MTA-STS, or config in the mail server, or many hosted solutions like Google Workspace also support enforcing this for selected or all domains).
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
I believe you're correct. I should clarify that the SSL cert problems I have are only related to my client connecting to the server to send or receive messages.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
I have used Mailinabox on a Hetzner server for about an year. My email delivers to all the major providers. However, small providers will occasionally block my email. So I continue to use my Gmail address for now.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
For me, the issue wasn't that I had to fix it frequently, but that when I did it was urgent, stressful and disruptive. Eventually the VDS I was renting had a fatal HD crash and I gave up.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
There are many benefits to running your own server. The three biggies for me are:
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
Anything that comes in from an address I have never seen before is handled specially. But it's not hard to filter out the obvious spam. Just a handful of heuristics on the from and subject lines (e.g. if the sender's name contains common English words it's probably spam) takes care of >90% of the cold calls. The rest I just look through manually once a day or so.
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
The maximum bound for most gTLD registries is ten years (have a dig around the ICANN website if you'd like to check), not fifteen. Sure, there might be a ccTLD that allows you to register a domain for longer than that, but I'm not aware of any, and ccTLDs usually have lower maximum renewal/registration bounds than the gTLDs.
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
There is no such thing as a BCC header. That's the entire point of the BCC.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
Typically, if you sign up for an email account, you get an email address like skywal@gmail.com or skywal@yahoo.com. Alternatively, if you own/host skywal.com, you can have an email address like skywal@skywal.com served from a computer in your home.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
So in the example of the parent, he's got a domain name registered somewhere (mydomain.com) and he set, in its MX records, the gmail server. But how does gmail make the connection between that address and your gmail address then?
So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host skywall@skywall.example with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, add our servers as your MX record in your DNS, or transfer the whole domain to us and we'll do it for you". In this case we're doing the 'changing the MX records' part.
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for skywall@skywall.example". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a user@gmail.com address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
delv MX ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com. 295 IN MX 20 alt2.aspmx.l.google.com.
ycombinator.com. 295 IN MX 10 aspmx.l.google.com.
ycombinator.com. 295 IN MX 20 alt1.aspmx.l.google.com.
ycombinator.com. 295 IN MX 30 aspmx4.googlemail.com.
This is me using DELV to ask "where should I send email for ycombinator.com?" and I have four responses. Column 5 tells me the priority. Lower numbers are higher priority. Unsurprisingly, this is Google. But let's see where they host their DNS, shall we?
delv NS ycombinator.com @9.9.9.9
ycombinator.com. 159148 IN NS ns-225.awsdns-28.com.
ycombinator.com. 159148 IN NS ns-1914.awsdns-47.co.uk.
ycombinator.com. 159148 IN NS ns-1411.awsdns-48.org.
ycombinator.com. 159148 IN NS ns-556.awsdns-05.net.
So AWS. So they have separate DNS to Email, and could change those MX records to host their email anywhere else, without needing to change or move ycombinator.com's DNS from AWS.
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
There are some special domains of the format <reverse-ip-address>.in-addr.arpa and <reverse-ipv6-address-digits>.ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
> The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
So, mandatory third-party doctrine/warrantless surveillance.
It’s a misnomer to say you can “own” a domain name. You cannot. The better word would be “rent”. Because of that you should think about what could happen if your domain name registration lapses and someone else registers it.
I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.
I think people who get their email on their own domain generally consider it as their permanent home and don’t think about what will happen when the domain no longer is registered to them. Using the apartment rental analogy works up to a point. With an apartment you can get the postal service to forward your mail to a new address. With a domain name there is no way to get this to happen. Whoever holds the registration controls the MX record and can do whatever they want.
In case of death it's indeed an issue if someone isn't planning ahead.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
Most emails are still going through servers owned by Microsoft or Google, so what does self-hosting email accomplish in reality? Some government entity likely has warrant-less access to most of your emails regardless. I like email as a way to communicate, but speaking pragmatically, it’s just not a secure means of communication in 2022.
My server uses encrypted connections and my own certificate. If I send an email to a family member on my server, no one on the Internet can read its contents. If I add a recipient and FAANG or Microsoft then yes it wouldn't matter. If that is truly important, we can learn to encrypt emails locally first.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
what does self-hosting email accomplish in reality
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
Using a 3rd party email host that isn't Google has the same effect with a lot less effort. I switched to FastMail for this reason, even if Google claims not to scan paid Gmail accounts for marketing purposes.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.