I would really say that "running your own email" is a set of things that can be done independently:
- Getting your own domain and using a provider such as fastmail or proton is a first step that gives you lots of security fom arbitrary. Because you own the doorstep, you can change provider without having to inform all your contacts of the move. You're also more secure from unilateral moves from your provider.
-Hosting your own mail server means that you are responsible for the persistence of your mail. It's a nice artisanal thing to do, and you may be satisfied to know that no one is reading your mail.
-Sending your mail yourself is the real hard part, because you need a stable IP that is accepted as a legitimate mail sender. Moreover, you need to monitor this property in the long term. Every mail server has their own way to choose who is a legitimate mail sender, and it's an ongoing pain to check that.
You're not forced to go all the way, you can simply pick a domain to secure the frontdoor, or you can host your mail server without sending mail by yourself, etc. You can also self-host, and change your mind later without much impact.
I personally would incite everyone to do at least #1 for safety reasons, #2 if you want to fiddle with the system to know how it works, and to avoid #3.
Internet was meant to be decentralized. Lets leave it this way.
I am running my server for 15 years and couldn't be more happy with "artisan" infrastructure.
I don't want or use webmail (sluggish), I don't want others fingerpoking my emails, I don't want various compulsory registration systems (like requiring my phone number out of """security""" reasons like google), giving others the ability to kill my account and do me a huge amount of work and on the top of that, it breaks the sites registration schemes, I have set up a script that accepts any email with some special structure and each and every registration gets a specially customized mail address (that I can calculate in my head, no configuration needed) that can be resolved back to the registration.
Getting spam? I am sick of you, whatever? No issue, just REJECT the whole address. It is used by only one site, like smart people don't reuse passwords, I don't reuse email addresses.
And you would be surprised, how many sites sell email addresses to others, and I know it as every one gets its own email address.
Rspamd eats the spam just as good as "ai infrastructure" /s
Even if you go for 3rd party email infrastructure, registering a domain is a must, so you can switch the provider fast if it gets vampirized.
Out of my whole infrastructure (100% self hosted, as said, for 15 years, actually more but not 100%), the email server is the part that needs the least attention.
The response to the author would be: nice that large providers have webmails and some other quirks that I don't want or need. Feel free to use them, but I have freedom.
The internet was meant to be used by people who knew how the internet worked. Herein lies the problem.
This might sound like gatekeeping, and maybe it is. When these systems were designed, they were not designed to be used by everyone. They were not designed to be commodities that are bought and sold, with the most valuable trinket available being the attention of the user. But this is where we are.
Few are capable of running their own _anything_ on the internet, and even fewer have the desire to do it, because if you run it well for yourself (as an individual), someone else will want you to do it for them because you are already doing it, so it's not that much more work, right? \s
Decentralization limits monetization of anything, so that is going to be a non-starter for investment of resources. Unless you are trying to have your infrastructure survive a nuclear war, no one is going to provide the means to build anything big unless you can sell it or the users of it.
The notion that anything really works on the internet with the assumptions that were made in the 70s and 80s, and the realization that what holds most of it together is the blood and sweat of ops, duck tape, and fever dreams consistently astonishes me. In the not so distant past, someone paid me to write them a custom FTP server. In the 21st century. It's like being asked to whittle an engine block out of a tree.
> Decentralization limits monetization of anything, so that is going to be a non-starter for investment of resources. Unless you are trying to have your infrastructure survive a nuclear war, no one is going to provide the means to build anything big unless you can sell it or the users of it.
I'll go further: centralised systems can emulate decentralised systems, but not vice versa. Thus, ultimately, the only USP of a decentralised system is that it is decentralised for the sake of being decentralised, and nobody cares much about that. Centralisation is inevitable, and wins out every time.
They can pretend to be decentralized, but they can't emulate the lack of centralized authority.
People certainly care enough about centralization once it's consistently abused in ways that hurt them (which always happens eventually, given enough time). Our existing anti-monopoly laws came about like that.
> And you would be surprised, how many sites sell email addresses to others, and I know it as every one gets its own email address.
So much this. I've actually contacted companies to tell them they've been compromised because I started getting phishing emails. I quit after the third time of reporting it and being told "we haven't been hacked, someone in your friends group has and you just can't read email headers".. right because someone in my friends group emails "mylocalgym.com@mypersonaldomain.com" to schedule group activities.. then six to twelve months later I get an email from HIBP telling me said website was hacked and my email was compromised.
It's funny to hear this experience. I've been doing this consistently for about 5 years now and have noticed 2 instances where this occurred, and in both there were prior disclosures about a security failure.
I tend to sign up for a lot of things (I'm seeing over 150 unique email addresses I receive emails from using this scheme), but I guess I'm just getting lucky.
Also, just out of curiosity, where does one sell email addresses, and how much are they worth? I take signups on a few websites, and I'd never sell my users' email, but I'm just curious to learn more.
I used to trade e-mail addresses with various banks, some 15 years ago. I'd just call up their marketing departments and offer those as 'financial leads'. I had a network of people who had various ad campaigns running where a customer could win something if the'd leave their personal (financial) data.
I did it in the early 2000s with a domain I picked up just for that purpose, which I'm almost positive was "myspamstopper.com", but I let the registration lapse and it was snapped up. It's amazing what was still available back then to easily register.
Side note: I've seen some MTA systems having weird filters for receiver's domain name or company name being part of sender's local-part.
When I'm opening an account at Example Bank which uses example.com domain, I avoid creating dedicated mailbox or alias with words "example" and "bank". exmplbnk@, xmplbnk1234@ or similar seems to have better deliverability when I'm attempting to contact the other side.
Perhaps there should be a system that lets
[1] ordinary people record that they notified a company that said company had been hacked together with timestamped evidence of said notification.
[2] people/organizations who sue/regulate said companies wrt said hackage have access to said timestamped evidence.
I don't know how to monetize said system but it would produce both social and economic value.
I've discovered two previously unknown data breaches this way. I was gratified when the operators of the sites thanked me for reporting it. Most times, though, I get the treatment you're describing.
Honest and non-rhetorical question here: Have any of your customers had an e-mail they've tried to send not arrive because the recipient's system was using a black-hole list that, for some erroneous reason, had you blocked? If so, were you able to successfully communicate with and/or reasonably work through whatever issue got you black-holed?
I haven't administered e-mail servers for 20 years, but back when I did, this started to be a problem that eventually became insurmountable. I used to manage a small business oriented ISP. We were multi-homed with a /18 that we used for everything. I had a customer that was a reasonably sized organization that dealt with tourism and conventions for a major city. On one of their websites, (which we hosted with IPs that came out of the same /18 as their mail server,) they had a directory of vendors who were associated with them. ONE of those members had a website that had been hacked/defaced. This got our entire /18 on a blac-khole list. They had an employee that was trying to send e-mail to someone on a system that was using this black-hole server to filter spam.
When we explained to them what the problem was, we got glassy-eyed stares back at us and a, "just fix it." I told them that, they would need to remove the link to their partner's site from their website in order to get them AND all of our other customers using numbers inside our /18 de-listed from this particular black-hole. They asked, "We have hundreds of partners who pay for membership in our organization and being listed on our website is one of the benefits. How can we possibly police every one of those websites every day to make sure there's no defacement or serving of any problematic material from any URL in any of those domains?" That's a decent argument in my opinion. And I tried to explain that different black-holes have different policies and no black-hole is demanding that anyone use their system for filtering. I tired contacting the organization that was using that black-hole to explain the situation to them, but they weren't interested in discussing it. As far as they were concerned it was our problem to deal with.
This kind of problem happened dozens of times with varying degrees of severity but with increasing regularity and it was one of the primary reasons we quit hosting e-mail and started re-selling another vendor's solution. That was a long time ago, and maybe black-hole lists aren't a thing anymore.
> If so, were you able to successfully communicate with and/or reasonably work through whatever issue got you black-holed?
Yes. Practically all black-lists have a de-list form that one can use, and most seem to auto-delist fairly fast as soon they don't get any more reports from honey-trap and other sources.
We do have a few custom written ways to detect hacked accounts, and we don't allow users to set their own passwords. We also tend to discourage/deny users who do newsletters and other "higher risk" form of email. All emails sent by websites is sent through different servers, which also mean that a hacked website does not impact the reputation of the email servers.
Events with black lists maybe occur once a year and as I mentioned above, fixed fairly fast. One good tip is to keep an automated eye on the mail queue and react quickly when things start to look wrong.
It's better now, but in the early days of organized blacklists (more than 20 years ago) it was somewhat chaotic. Many large ISPs ran their own blacklists and some were poorly managed. AOL was the worst of them all. Their admin staff was unprofessional and unresponsive when I provided a PoC for their defective spam control system.
There are a few sites where you can plug in an IP address to see if it's on any blacklists. A handy thing to do before setting up a new server is to work with your provider to find a clean IP address beforehand. Here's one that I have used: https://mxtoolbox.com/blacklists.aspx
Thanks for the link, I see my email server is on a few lists. I’ve heard that DigitalOcean isn’t good for email servers but I’ve been lax on investigating.
Now I have incentive, this is annoying. Maybe I need a static IP as well.
For what it's worth I've run into the same set of issues at corporations using Google and Microsoft's hosted offerings. Hell, sometimes you can't even send it mail between customers!
> Have any of your customers had an e-mail they've tried to send not arrive because the recipient's system was using a black-hole list [...] ?
Yes. Twice.
In the first case, the mail provider was our ISP; and they got themselves in some mainstream blacklists. The problems getting that sorted out were part of the motivation for bringing mail in-house.
In the second case, there was some academic departmental mailserver and they were using some list incorrectly; using an extremely-opinionated list to block when it should at best be used to score.
This wasn't in itself a big deal, but one of my boss's correspondents was a senior professor in this department and they had some important business; and the postmaster was a dick, and wouldn't help. Boss didn't want to use some secondary email address; I had to show him how to set up an alias on some commercial server, which was second-best, but he was in a hurry.
Boss was angry with me and barked at me. If you run a mailserver for some group, one you assembled yourself, then people expect you to take responsibility for sorting out any mail problems. Well, they're right: you have taken on that responsibility. You made it, and you're running it: who else can they complain to?
[Edit] My point is that it's not hard to set up an artisan mail system; what's hard is that you create a job for yourself that is at the same time networking, user-facing, and technical. It's an interesting learning point, and I recommend it. But don't underestimate what you are taking on.
> [Edit] My point is that it's not hard to set up an artisan mail system; what's hard is that you create a job for yourself that is at the same time networking, user-facing, and technical. It's an interesting learning point, and I recommend it. But don't underestimate what you are taking on.
This. So much this.
I will happily run my "artisinal" mail system for myself. Would I put customers on it? Oh, hell, no.
I, sadly, always recommend that companies pay money to Microsoft for email. You are really paying for the customer support service rather than the email service.
Microsoft's email behaviour seems like anticompetitive abuse - you can whitelist an address and they overrule you and block incoming email for obscured reasons.
Customers still have problems occasionally _sending_ email to one domain, which is over 15 years old and sends <1 email per day. If they initiate and email us we can't send them a reply (if they're on MS email, sometimes). We use an outlook.com email nowadays as a relay and have to treat MS using customers differently still despite using a relatively large supplier.
Some years ago, I was lead to believe, you could pay a third-party to add you to what was effectively MS's whitelist.
Aside: back then I was doing some webdev and supporting IE5+ so I already hated MS about as much as one could.
Never paying to enable interoperability that is part of being a reasonable web citizen/company. Paying them just reinforces the negative behaviour.
Please don't recommend to people to pay money to have all their email communications read and stored by Microsoft, the US government and possibly other parties.
There are plenty of other email providers which are worth considering, and I'm sure some of them have half-decent customer support.
It’s not so much that I want to go all the way and do it myself, but I’m interested to see the gold standard way and as per the OP, perhaps go part the way (ie not send) - do you have a resource you could point me at that you recommend or rate?
Not trying to get you to do my dirty work, just wondering if you have a resource you use.
Serious question: what if you're on a network that blocks non-HTTP[S] traffic? How would you read your mail? This is a problem I hear of quite regularly.
> What are the memory requirements for postfix, dovecot, and rspamd these days?
Approximately nothing. I run all my email infrastructure on the smallest available $5/mo Linode and it's way overprovisioned even so. I'd take a smaller VM if they offered one.
Hmm. Admittedly, I didn't use postfix, but dovecot was the largest process running on my 1G server and adding rspamd resulted in it doing the Out-of-memory mambo a couple of times a week. This was just for my domain, but I do get a crap ton of spam.
Doesn’t even need that. I’ve run pretty much that stack on an original Raspberry Pi. You wouldn’t want to be trying to do a lot else, and GB sized attachments are gonna be limited by the usb Ethernet speed, but it’ll run just fine.
FWIW, someone is still reading your email. The owner of the infrastructure of your message recipients. If you've gone through the trouble of hosting your own email to avoid Google harvesting your messages, but you're exchanging emails with someone whose email is hosted by Google, then your emails are still being harvested and/or are harvestable.
Good point. But it is still one person/company less that reads your e-mail, if you and your correspondent used different e-mail providers previously.
There are also correspondences where you are only the receiver. For example, when you order things online. Gmail doesn't need to know what you are shopping for.
Unfortunately, they can still close the loop on purchase history and ad impressions through indirect, lazy association.
It's a very, very impressive edifice that's been created for identifying and tracking pretty much everybody irrespective of their direct, immediate interaction with the entity doing the tracking. I honestly think it's kind of funny that such a potentially insidious system was manifest for something as principally vapid as ad targeting.
True. But at least, you went out of your way to not inflict this to your recipients, should they wish to avoid this harvest. You made your part of trying to avoid this. You did your part to weaken this argument. Someone has to break the vicious circle.
There’s a midpoint at #1.5 where you control the domain and rely on an external host, but also have a continuous archive of your historical mails on your own server. Otherwise with a standard IMAP setup, if your provider locks you out, you’re limited to only the most recent N messages on connected devices.
Buy a domain for $20, get a GSuite account with it for $6 a month or similar, then archive all your mail via Outlook or your desktop email app of choice. You also get a lot of other tools and storage space for very low cost which I find useful.
It's still using Google but it allows plenty of control and management, and I can take my domain anywhere with minimal stress. It's a decent compromise.
Just make sure that you use a different domain registrar than Google Domains, and make sure your admin email address is not gmail. Because if Google bans your account, you will be totally screwed.
I used to do this but switched to iCloud Beta for custom domain this year. It’s $1/month (free with any paid iCloud plan) and integrates with iPhones/Mac, supports push email for iPhone, and is free of Google spying.
you could argue dmarc, dkim, and SPF were all invented as barriers to entry for small mail servers as the majority of internet SPF is permissive and the majority of DKIM is misconfigured. imo, senderbase and other reputation/policy proprietary shit-lists used by anyone with a Symantec or Cisco email product only served to convert the unwashed to a corporate license as most of these mechanical turks just paid the same spammers by night to show up in a delhi office complex day job to identify their own campaigns. DMARC and DKIM/SPF just wrapped email in a mandatory layer of arbitrary complexity to "solve" a problem that RBL's had largely managed to tackle as an independent entity.
that they exist in part to force the hand of small companies and users to simply submit to a big player for their email is something i have long considered.
SPF/DKIM aren't meant to solve spam directly. They are aimed at sender forgery.
The real benefit of SPF is for outgoing mail. People no longer forge my domains. It stops backscatter. It has almost eliminated the mistaken spam reports to my ISP by people who don't understand mail headers.
Huh? SPF, DKIM, and DMARC along with mta-sts/DANE require running some scripts and entering DNS records. If you are already capable of hosting a mail server at your own domain, then this should be rather straightforward.
I'd like to see something to make it easy to address the case where you are doing #1, but your provider does unilaterally kick you off with short notice for some reason. The email equivalent of a bug out bag [1].
This would be something that provides in a single package an SMTP server, an IMAP and POP server, pre-trained spam filtering, and maybe a web server with a web-based email client, and a simple setup program that asks a few basic questions such as your domain name and configures everything on your end and provides help for configuring things elsewhere (such as with DNS, such as telling you what to put in your SPF and DKIM and DMARC records).
This is meant as something to handle your mail during the time it takes you to find another provider. It is meant to be something you can quickly install on a VM somewhere, point your MX record at it, get a Let's Encrypt or similar certificate for it, and not be losing mail while you are between providers.
It should have a quick start guide that includes details on signing up and getting a Linux VM up at major inexpensive VM hosting places. Amazon Lightsail, Hetzner, and such.
It should make minimal assumptions about your Linux environment. Probably it should not use the SMTP, POP, and IMAP servers that are packaged by your Linux distributor. It should use minimal servers that are written specifically for the emergency mail kit.
I have my own domain since when I was about 15 years old and used that for a while on a digital ocean droplets. It's incredibly easy to set-up.
The only issue I had was that other people were not getting my mail and I sometimes it was not even reaching their spam folders. Probably because Google/Microsoft were blocking that IP range of Digital Oceans.
Nowadays I just pay for a personal Gsuite license and use Google Infrastructure.
Much simpler that way and I'm almost guaranteed that my mail will reach the recipients. You only need to set-up your DMARC / SPF records and point your MX records to the ones that Google provides.
We've been running a small mail host for ~10 years (less than 100 accounts, but outgoing monthly newsletter to a few thousand addresses) ... we had basically zero problem with delivering to Gmail. Their spam filter, while strict and applies throttling (and a bit of greylisting) is completely livable, compared to the balls-to-the-wall insane Outlook/Microsoft "protection" ( https://news.ycombinator.com/item?id=28982434 )
If you put an address or domain in the safesenders list; they do literally nothing. Like you can just totally spoof the domain entirely.
However if you use transport rules as per their rec, there’s all sorts of stuff that will still get flagged, and you have to to reference ATP, anti-phishing, anti-spam policies. Much of which aren’t even in the Exchange admin panel, rather they are in “security” and buried in hamburger menus galore.
And what’s best. They don’t even have any documentation for how these modules interact or what order mail is processed in. I had a case open for months thst finally got escalated to someone that was able to explain the issues we had with specific list serves/domains getting flagged.
In the end my only option was to whitelist emails classes as phishing and route them to junk rather than keeping them in quarantine. Even though it was a 99% accuracy rate sans this single domain.
The guy was really only able to commiserate with me. We are but a number and not a big enough one to get Ms to change a thing. Their best recommendation was to deploy an edge device like proofpoint/proofpoint hosted and just handle it from there.
I get what they want to do. They are trying to make the crazy email RFCs easy for devops guys thst don’t give a damn about how e-mail works. But it’s still hard to keep up with as they constantly just move stuff around and change their own standards on a near monthly basis.
Well....that's how I found out about it when I took on my current role. We had pretty solid phishing attempt slip through. I was able to spin up a VPS as test it on mine and some other known tenants as well (with their permission). And since o365 uses a predictable name for their SMTP receivers for a tenant (domain-com-net-whatever.mail.protection.outlook.com)its easy to kind of....select targets and test it out.
So even if its not listed on the domains MX record but you can suss out they are an office365 tenant receiving mail, you may be able to relay off it and spoof to high heavens (especially if the edge device reccomends you....ahem...whitelist your own domain and not use transport rules). In fact especially if you can do this.
For example i think MS forced proofpoint to change their config recommendations as an outcome.[1]
from the page on [1]:
"Due to major complaints, Proofpoint has opted to change change to the format of ensuring Proofpoint mail is not scored via the O365 system. This rule will allow external email to come in still, but will follow O365 scoring. This is to ensure no mail is lost."
I've been running my own (and other) email servers for over 25 years. About four years ago I switched mine over from sendmail (with a bunch of add-ons like spamd/spamassassin, rbl, etc.) to mailinabox. Mailinabox is full-featured, secure, and reliable. It doesn't take anywhere near the level of effort required to maintain vs. other solutions.
Microsoft has blacklists the entirety of Digital Ocean and won't whitelist any IPs, even if it's a legit mail server. If I didn't know better, I would say that's anticompetitive behavior.
If true, that is probably the last excuse I needed to migrate my own email server off Digital Ocean to another provider. Oh well, it's been a very good run, DO.
I use a domain, registered at Namecheap, and I forward to my gmail account. If gmail “goes away” I simply configure my email to forward elsewhere. If I’m unhappy with Namecheap forwarding, I point my DNS at another forwarding provider. If I’m unhappy with Namecheap, I transfer my domain to another provider.
It gives me all the flexibility I need with almost no work or maintenance.
There are enough mail providers that I could easily switch to that I don’t need a piece of software. Switching from gmail to yahoo, proton, apple, outlook, or juno is a simple domain adjustment and has me back receiving mail within the TTL period.
If they are using GMail for work, the DNS just points to GMails actual server and authenticates using DKIM as well. Google for Work will provide you with the necessary DNS entries to set. Obviously this will not work with their free offering, you'll need to fork over $6/month for this.
Yes, though this isn't forwarding. The mail is actively being delivered into gmail, who must be programmed to accept mail for that domain.
Forwarding, on the other hand, made it possible to do a simple redirect of one address to another (eg. ~/.forward) but strict SPF rules will deny the forwarder as a valid source for the mails.
I used to do this but found every now and again I'd get multi-minute long delays before gmail picked it up. I've since moved to paying for Google Workspace to host my email (eventually hoping I'll move to something cheaper).
The goal is to get my personal domain to be my email domain for exactly this reason.
For my soon-to-be-born son, I'm registering him a personal domain immediately, and I'll turn it over to him when he's old enough to have email - save him some trouble.
As someone who de-googled about 5 years ago by buying a nice three letter .io address this hits right in the feels.
I could go through the process again (not fun) with some ridiculously long .com/.net or other OG tld which are probably somewhat more resistant to rent seeking practices like this or I just suck it up and hope it remains pricey but not egregious.
For personal domains, I bite the initial cost and buy the domain for 10 years, then every year top it up to 10 again. For a $20/yr domain that’s only $200 up front, and if the cost suddenly goes up or some other TLD policy changes that I hate, I have plenty of time to gradually move to a cheaper/better domain.
One thing I’m not sure of is what happens if I want to switch registrars in that time—will the full 10 years of ownership transfer to the new registrar?
Like you, I have domains registered for 10 years and extend it every year so that it always remains at 10 years.
If you switch registrars, your domain validity continues as before. Your registration information is with the registry for your TLD, not with the registrar alone. So your 10 years of (future) ownership will carry over. Two caveats to note. The first is that you can’t transfer a domain within 60 days of purchase or renewal. The second that I’ve seen is that a transfer to another registrar requires a one year renewal for the domain. So I’m guessing you may not be able to transfer a domain that’s already at 10 years (even taking into account 60 days after the renewal to keep it at 10 years). But if your domain is at nine years, you’d surely be able to transfer, and it would become 10 years at the time of transfer.
The expiry date never changes through all these actions.
Epik lets you buy domains more or less permanently for a large upfront cost (~$600 for a typical otherwise $10/year domain IIRC). Anyone know any other registrars that do?
Indeed. I set up my personal email server on a Hetzner VPS using iredmail. It just took a couple of hours and everything actually works pretty well with very little maintenance. Even Hotmail/Outlook.com accepts my emails.
I think #1 is a super solid idea. I'd love to go beyond that— I'm familiar with the tech and love the satisfaction of a more DIY approach— but other end users preclude my doing so.
The author mentions quality in big email service but only passingly mentions what that encompasses. Smooth, responsive, well-worn, ceaselessly preened, and smoothed-over end-user UIs are important. Unfortunately, the open-source alternatives are comparatively rough.
(As a long-time developer and more recent designer, I write a lot of open-source code myself. I understand that these are complex and tedious problems to solve. However, without frank critique, "Open-Source Alternatives" will always be "Alternatives.")
Every interface I saw needed fundamental design work. My recent research showed 2+ decade old interface layouts w/new features just bolted on, visually complex toolbars, menus, and lists, little editing for views and controls, and comparatively unattractive designs(, which even if it doesn't matter to you, that doesn't invalidate its importance to others.) Even this crowd— people accustomed to configuring complex applications— lament the clunky interfaces.
To me, most open-source interfaces are like eating on a diet. Your sense of accomplishment offsets the discomfort... at least for a while. End-users, however, don't have or need, that holistic view of the service. To them, the interface IS the service. DIY/tech accomplishments are abstract and indirect factors, at most. For most, it's like eating on-diet, but someone else loses weight. Attractive alternatives make that unsustainable.
So the real hard part isn't technical— it's assembling an email stack where users don't feel deprived for having chosen it.
The solution is more collaboration between design and development expertise within the FOSS. If you have a position of authority in any FOSS projects, I implore you to be open-minded when presented with interface design ideas.
Happy to talk about productive ways to engage with designers and design feedback.
Firstly, I am one. I have ten years of full time back-end web dev experience and other types of coding for over a decade before that. I also know others— UX designers generally start in another field and dev work is a pretty common start.
Secondly, that few contribute as designers rather than developers is definitely a chicken and egg situation. Designers time and effort is universally seen as less valuable than developers’ and therefore more readily dismissed or minimized. People are worse at taking critique for things they’re not confident in, and as you note, most open source projects are maintained by developers. Ever give a brand new developer a code review? Yeah. That’s about what it’s like critiquing an open source project’s beloved “quirky” interface.
I’ve seen eager designers post issues in repos— some with complete wireframes and rationale having done a good amount of work already, asking for specific types of feedback— only for their system to be instantly bikeshedded into oblivion rather than productively discussed. Unsolicited contributions are often viewed as superfluous expenditures of dev time, or even viewed with outright suspicion or hostility. If it’s not submitted in the form of bite-sized PRs ready for production with the understanding that existing devs can veto any changes without any real justification. Going from a haphazardly assembled UI to a properly designed UI requires fundamental change, and that’s a lot of work. Would you contribute code in a project with those competing requirements?
Before any of that, any designer interested in open source software has almost certainly made the mistake of griping about the interface for gimp, or git. It’s a good preview for what lies ahead.
> Sending your mail yourself is the real hard part
It's possible to punt on this by using SES for outbound, while continuing to handle inbound a different way. Obviously SES doesn't count as fully self-hosted, but it does solve (or at least significantly ameliorate -- zero issues here) the reputation problem
I think that's what they're talking about in terms of 'you don't send your own mail', but it is a good solution and SPF+DKIM means you should have very few issues with deliverability and reputation.
I mean that SES gives you SES IP reputation, and they force you to have a low bounce rate & complaint rate, thus SES specifically is usually not blocked by big providers nor small enterprise networks.
I was determined to make #3 work for years. But despite a golden reputation for my IP, perfect dkim, dmarc, dns, and everything else, plus exclusively personal mail (no bulk mail ever) I could never get out of the spam box at several major providers. Never could figure it out, even with all the tools. Finally gave up.
I have come to suspect new MX servers are spam-holed by default until enough people click “Not Spam”, which is an absurd hurdle for a single user hobby server.
> I have come to suspect new MX servers are spam-holed by default until enough people click “Not Spam”, which is an absurd hurdle for a single user hobby server.
Yes, a fresh (or: previously sending spam) IP requires some warmup time until providers like GMail will let you anywhere near the inbox.
And if you're not sending out a high enough volume of emails, no chance.
I had the IP for probably five to seven years- but my outbound mail volume was less than ten emails a week. Which, of course, was tough to increase when delivery was poor - chicken and egg.
I monitored all the blacklists, filed ownership attestations with receiving domains, the whole nine yards. It’s sad that a microscopic MX server can’t be default trust instead of default spam for the first two messages a week.
(I want to repeat, this was a single user exclusively personal domain. Writing to a friend, to grandma, to a colleague)
I was probably in a bad ASN, but at that level trying to find a good block you’re just rolling dice. I wasn’t willing to play anymore.
> Sending your mail yourself is the real hard part
No, having quality spam and fraud filtering, and quality security, that you host yourself, is by far the much harder problem. I would argue that outsourcing your email to Proton or Tutanota is not running your own "artisanal" email server. By the way, even with those email providers, I still have terrible spam and fraud emails getting through filters that I never would have seen with my GMail.
Well, on the flip side unless you constantly check your spam folder (which pretty much completely defeats the purpose of a spam filter) you most definitely have lost important mails thanks to gmail.
I just don't see what people see in gmail part from the google brand - which surely isn't a good thing anymore.
Most people I know? It is almost a daily occurrence. Including if senders are in address books, and "not spam" is clicked when found in the spam folder.
Most people I know see legit emails in the spam folder, all the time.
Google workspace sends yes/no meeting confirmations that you make in workspace, to people within your own organization, to spam unless you specifically make a rule to allow them.
I still have a @gmail.com account and every time I check it there's a whole lot of spam sitting in the inbox waiting to be classified as spam. Very strange as my main address (which I've had as my main address for almost 20 years) is on FastMail and hardly gets any spam despite the address being much more exposed e.g. it's found on many public mailing lists and it's been part of more data breaches than I can count. Yet it's Gmail that gets the torrent of spam including many obvious ones ending up in the inbox. YMMV of course but I don't rate Gmail highly compared to FastMail and even Office365.
> I would argue that outsourcing your email to Proton or Tutanota is not running your own "artisanal" email server.
That's not a claim I'm making. The goal of my message was merely to help people see that there are several steps you can take from using a gmail account to relying on no third party. When I talk to people, they often don't realize this, and especially, how easy it is to set up #1.
> No, having quality spam and fraud filtering, and quality security, that you host yourself, is by far the much harder problem.
Handling spam is not an easy problem, but it's one where you have all the cards to take actions. On the other hand, having your mail properly delivered is something where people have wildly different outcomes, and for people with bad outcomes it's "impossibly hard, and there's no action you can take about it, unless you personally know the right people at the right places".
Might be a silly question, I own a domain that is my first initial + last name dot dev for my portfolio. What's a decent, or usual, prefix to use with such a domain as a personal email for job applications? Bonus point if it is English-French bilingual.
contact@ sounds off, like I'm a corporation. email@ or mail@ I kind of like, but I'm afraid it sounds "confusing" (is that in my head?). application@ or job@ is not bad, but a bit specific and not one I could use all around.
But when I give the mail directly to a given company, I use companyname@mydomain.com. That lets me track how I'm contacted, and sometimes it starts interesting conversations.
I’ve tried a bunch of pricacy-focused email services and have been let down by one or more aspects of their service. Pretty much all of them managed to handle sending Ronny satisfaction, though.
So setting up inbound to run on my own gear and paying a couple bucks a month for others to deal with dkim and and domain keys and all that other crap… that’s brilliant.
This is great advise. I do #1 and #2 but not #3.
I use sendgrid.com for #3. They have one of the highest (if not the highest) deliverability rates in the world and mails arrive really fast (faster than gmail).
As a bonus: I get to see report which of my emails were classified as spam or not opened.
Also, first 100 mails per day are free (which has been enough for me so far).
#2 has practical reasons as well, such as security and privacy (yes, other mail servers on the internet can catch your mails in flight; that’s quite different from a mail provider having full retention of your email at any point in time)
I’d add another thing:
- Hosting your own mail client. You can self-host roundcube/mutt/thunderbird/or even an imap server that just fetches (and possibly deletes) email from the remote server using something like mbsync. This mail client/server doesn’t need to interact with any other mail server apart from the mail provider that receives the incoming email, be that gmail or fastmail. While paid ProtonMail can be used for this, it’s a bit of a hassle with their lack of native imap support.
> other mail servers on the internet can catch your mails in flight
This also shouldn't be a problem most of the time if your email server supports TLS; Google currently sees 81% outbound email encryption[0], so you can imagine roughly 4/5ths of email servers support it.
Indeed. It was more of an inb4 of the common reply of “there is 0 privacy gains of self-hosting email since most of the people you’re mailing with will be on one of the big providers anyway”, which is tired and defeatist.
Tired, defeatist, and giving way too much credit to the big guys. No matter how good they are at assembling a profile on me from indirect data, it's still going to be more effort and likely lower quality than if they had a giant store of data labelled as mine.
I’ve long been interested in self hosting, but constantly tweaking and never got #3 stable. After iCloud made it easy to do #1, I pushed my parents into using it and gave them my domain (@lastname). Now I just share it too, since it’s just too easy.
I encourage everyone who wants to change to something they control, or uses a paid iCloud tier to set up their own vanity domain for iCloud. It’s so easy and lets you own the identity, which is a critical part and portable. Not trying to shill anything, but it took 10 minutes and is offered by apple, so every not tech-savy person has heard of and trusts them, so it’s easier to convince others.
Yep! I do #1 despite researching controlling the whole stack. I still like the idea of doing it someday, if only with a development domain.
I pay $12 a year for email hosting, $10 a year for the domain. I use name.com and I presume (though I have not tested) that if I needed a human to talk to, I would have much better luck than with Google. I also don't have to worry about a snarky Youtube comment getting me locked out of Youtube, Youtube TV, Gmail, GDrive and everything else.
You can have the best of both worlds with self-hosting received mail and achieving good deliverability by using a service such as Amazon SES. SES will probably cost you less than $10 a year for personal email sending volumes. I use it for my business and it is less than $15/yr. Rarely get a bill for more than $1. They hold you accountable for any abuse/complaints, which is a good thing.
I use it for personal use, would also recommend. It's not 'self-hosting' of course, but that's not what I actually care about personally, more interested in 'running my own' regardless of whether it's physically my hardware or not.
(Or rather given everything I've read about self-hosting email, not regardless, this is my preference...)
For a private mail server operator, Amazon SES is somewhat annoying because if your mail server is down, they're only keeping mails in the delivery queue for something like eight hours or so, which is way too short if you're not a big provider who can commit to round-the-clock support for the mail server.
How do you use SES for this? It seems to only allow receipt of email conveniently. Sending seems to require using an api which is not something most people want to do to email others.
Yeah, I've gone with the compromise of my own domain and a rented managed server for the last twenty years or so. Works well enough for me. I like to be able to give different email addresses every time I need to register somewhere. Keeps my personal address free of spam.
Someone should try "artisanal email server" using cloudron or yunohost !
the bigger problem is that "authoritative" email monopolies such as Gmail, 365 and the other big ones arbitrarily define and impose what is a legit email server or not and even with better score than gmail an "artisanal" email server can suffer from being classified into spam by the big tech players just because they can and will do anything to maintain their monopoly.
I'm running an email server and I can tell you that this is by and large not the case.
If you put some decent effort into making sure that you don't send spam, try to monitor if anyone thinks you send spam and react when someone complains that you send spam (and stop it), it works.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
I've run my own email for decades and I've designed and run some pretty big commercial installations.
As a small provider, you run the risk of existing in a netblock used by other people sending spam. A small co-op I ran encountered this problem once. They were operating on the cheap and while they weren't sending spam their neighbors had been.
Even as a large provider at a billion dollar company, figuring out delivery issues is a huge pain and generally not worth it. There are unofficial professional postmaster meetups around the bay and these can be helpful in getting escalation contacts to fix issues, but even with entire teams of people dedicated it's a lot to handle and is usually worthwhile to outsource the work to other companies who already have these types of relationships established.
Please stop spreading falsehoods. If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
I did run my own email server for 20+ years. As you may imagine, I had to learn a thing or two about DMARC, DKIM and SPF, but spread over the years it is not a big investment to make.
Most of the time, delivery problems were of my own creation. Like running out of disk space or accidentally disabling TLS.
Once in a while, Microsoft would start swallowing emails or Google would push everyone to use DMARC.
But overall, the experience has been very pleasant. I host my mails, I own my data. I am not shy of using Google, but my work is not defined by their whims. When Google tells me I ran out of space in my account I just delete stuff because I have copies of everything outside of Google infrastructure.
Lots of sister comments here saying that they've been running a mail server for X amount of years, where X is a rather large number. That will obviously come with some reputation for your mail server, reducing the curve of being classified as down. I would be interested in hearing from someone who tried to setup a new mail server in the last 1-2 years who was able to run it without a hitch.
In my case x>10y for a personal server, but that reputation got ruined when some test email account I had created with a weak password and forgotten, got breached and some spammers started sending spam. My mail server (smartermail) notified me within an hour of the abnormal number of emails and I disabled the account immediately. But that was it for the reputation of that IP. Fortunately I could switch to a spare, clean IP.
That being said, now I monitor and auto-ban failed authentication attempts to smtp/imap (among others) and running the service is fairly low maintenance.
But the morale of the story is that you are only one weak password from one of your users away from your mail server getting blacklisted as a spam server. So while I think it is fairly easy to run a personal server, running one for a small organisation is another matter.
According to AWS, most mail servers will not even count statistics for low-volume senders. If that is true, then it won't matter whether your personal email server has been up for 20 years or 1 year, it won't have any reputation.
Yeah, I have that problem with gmail, I had a test account with a weak password get exploited a few years ago. Now for any new gmail address I want to send to seems to endup in spam. The problem here is there is no getting out of jail easily for low volume email users.
My personal gmail account is full of spam and emails I do want from email lists end up the spam folder randomly.
This matches my exprience. I switched recently after 10+ years, and was cautious that this might be a problem but it hasn't been at all. I think it has more to do with the choice of ISP.
Interesting, I misinterpreted what was being said!
I'm doubtful a default block would work, as that would even penalise the 'big boys' of email when they make basic network changes and piss off existing customers of both sender and receiver... Its easier and logical to conclude something without reputation yet is therefore sending too few mails to be useful to a spammer.
I've had good experiences with smaller ISPs (currently Mythic Beasts). In contrast, OVH was a poor experience.
I find that reputation (beyond the known "block-lists") appears more likely being tracked for the whole AS number, therefore a lot more to do with your "neighbours" than anything else.
What matters most is if the IP address they issue you has been blacklisted for spamming. DigitalOcean is fine but you need to check the IP address before you do the work of building a mail server. Some of their IPs are on a lot of blacklists.
If it's only on a very few you need to look into who's blacklisting it. There are some that offer a way to get delisted and make it easy, there are others that block pretty much every IP address DigitalOcean has (or large ranges of them) and they won't de-list anything within them. Many of those blacklists are managed overseas and not used much in the U.S.
No matter the ISP you should check the IP address they issue for a VPS before you build the email server.
I've been running my own email server for around 7-8 years and just setup a new email server on a DigitalOcean vps earlier this year using "Mail-in-a-Box".
That's about as easy as it gets but it still requires some work and you need to check the IP address DigitalOcean issues to see if it's blacklisted before you set it up.
Google makes it easy to get whitelisted. Microsoft email services (Hotmail/Outlook) are a pain though. I tried to get through their process but got nowhere. Other services I had to submit a request to get de-listed. So it does take awhile to go through all that.
Still, I prefer that to hitching that wagon to a 3rd party provider like Google, or any other.
Before I set mine up the 1st time I'd been screwed a few times by 3rd party providers. The last one, I can't recall which, but it was either "MailChimp" or whomever bought them, that I'd configured an app to use and almost as soon as I'd released it they announce they'd been acquired and I would have to use the new services APIs, and of course they cost more, and their services were geared towards mass mailing, and that's not what my apps do, and their API sucked for my needs.
It was about 12 years of dealing with 3rd party bullshit that motivated me to set up my own email server.
If you just want to fiddle around with one to get a feel for it Mail-in-a-Box is a good place to get started: https://mailinabox.email/
I setup and have been running an email server for around 20 domains for over a decade. There have been no issues delivering to Gmail or Outlook, AOL, or Hotmail. There was some work I did initiallity to remove our IP addresses from blackhole lists, which had resulted from whatever the prior owners of the addresses had done. That was, however, minor and didn't take much time. Similarly, setting up DKIM, SPF, and the like were necessary and ugly to do, but they didn't take much time.
My mail server running on DigitalOcean has been relatively trouble free over the last 9 years. It runs docker-mailserver and is used by me and a dozen employees of my various small businesses.
It requires some effort to maintain and understand, and I’ve had a few deliverability issues over the years but they are generally with niche providers. I’ve never had trouble sending mail to the big providers.
Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I think DO is really good about policing their IP space. When I signed up for the Microsoft JMRP [0], DO was already a contact of record for the IP I was using. I just appended myself to the list to get any abuse reports as well.
>Every time I read comments about the impracticality of self-hosted email, I scratch my head. Maybe I’ve just been lucky.
I feel the same. I've had one or two hiccups but smooth sailing for the most part. I'm also happy to provide receipts that show how the recipient's mail server is responding when I send the emails. It's a powerful tool to say, "your mail provider is misbehaving, look!" They will wonder how many people tried to send them email that didn't get to them.
That's what I've been using for some years and it's never been a problem for me. You are right that you have to have at least a basic understanding of how a mail server works, and there is some configuration to know about. But I think of all the things I host myself, docker-mailserver is the least cumbersome and among the most reliable.
I've been running my own mail server since mid '00s. Initially hosted with one of West coast Canadian colos and subsequently moved to an EU colo. Had some deliverability issues with Outlook and Yahoo, but these were episodic and rare even though I set up DKIM only last year and have been running with just SPF and DNS/PTR before that.
I know at least a dozen of others with similar setups and timelines. But we all use dedicated colo'ed boxes on IPs from clean netblocks that weren't previously used for shared hosting. I strongly suspect that attempting to run a mailserver on Digital Ocean, OVH, 1and1 and similar mass-hosting providers will not go well. Just like it will be an uphill battle to run it on a residental IP.
What did I generalize, exactly? Parent poster was claiming that anybody can set up a mail server with good deliverability - that's a generalization. I said good for them (acknowledging they managed to make it work) and said that I also tried and couldn't make it work - therefore, clearly not everybody can make it work. Did I not argue against generalization there?
> If you were able to somehow get your own email server to deliver email to Gmail and Outlook, great, good for you - but stop pretending that anybody can do it.
Yes, that's probably true. I've been running my own server for 20 years now, and I guess that in itself helps with getting my mail delivered (apart from t-online, but who cares about them). At some time I also hosted some mailing lists, but I quickly abandoned that because that's a surefire way to get your IP blacklisted sooner or later. If you set up a completely new mail server, there probably is a lot of luck involved, and I wouldn't recommend it to anyone, at least not for your critical business mails. I pretty much keep doing it only out of nostalgia, it doesn't really make any sense otherwise.
Haha I have the same experience... I have given up trying to send emails to t-online, but every other email-provider accept emails from the server I manage. It sends a few thousand emails per day.
A few years ago we had problems, but then I realized some of the emails sent from our servers had non-ascii characters in headers (subject, from, to) which caused email-providers to distrust our server. Using encoded-words syntax ("=?UTF-8?B?" + BASE64(text) + "?=") fixed that problem:
I think the "decent effort" part is the key thing. We had to change our mail routing temporarily earlier in the year (after having sent via Office 365 for multiple years) and keeping on top of emails that were being blocked was a non-trivial amount of effort (and stress) for a period of time.
Unlike the person to you're replying to we had no issues with Google or Microsoft (once we did the requisite things) - it was Yahoo (and the people they provide email for) and then multiple mid-size organisations who used IP based block lists. At one point our mails were being rejected by our local NHS trust, the London Fire Brigade and a mental health agency we make referrals to. None of this was complicated to resolve but it was energy that could have been better used elsewhere.
I'm not usually part of the "let's go cloud without doing any cost-benefit analysis" movement but with email delivery I was happy when we could go back to routing via Office365 again. If a recipient decides to ban Microsoft's IPs that's usually going to be a bigger problem for them than me.
I've been running mailservers using free software for 20 years. I've run two for personal use, and several for groups like companies. In the old days, you could indeed throw up a server, and provided you don't spam, and you're not in a bad neighbourhood, outgoing mail would be accepted.
In more recent years, my experience has been that it takes time for a new mail sender to be acccepted; could be a year or two to build reputation. That's assuming you do everything right.
My personal mail, by the way, has been on the same domain since about 2001. I've quit running a mailserver now. My small ISP runs a setup that's basically what I would have built, so I use that; the support is excellent. But it's still on the same domain.
Last company I was at ran their mail on their ISPs mailserver. The ISP got taken over; service deteriorated, to the point it became unacceptable. So I built $EMPLOYER a mailserver; it took me longer than I predicted, because the bosses had all kinds of finicky requirements (don't they always) that I had to figure out how to provide after the fact. But that "artisanal" server beat the bejabers out of the ISP system; it was fast, reliable, and when anything went wrong I could fix it - which that ISP couldn't.
To most, including Gmail, it's actually no problem with DMARC in my experience too.
However, one of my servers IPs is on a Microsoft blacklist since many years now.
It sends <10 messages / day. I've tried every unlist form I could find, even called MS but it does not get taken of that list and they "won't disclose why".
I'm routing SMTP to MS via another relay now :)
relay_passwd.db: # if necessary / not authenticated by IP
relay.server.tld user:pass
The relay can/should rewrite the Return-Path to pass SPF. It's no problem for DMARC as the DKIM signature added by the initial server still authenticates it.
It requires manually adding domains of custom 365 installations to the list - at this size I do this manually, but should probably be automated "on bounce" or maybe even by a smart rule based on the MX record.
In Exim4 it's also possible to conditionally rewrite based on for example the recipient domain.
You need control of the entire netblock you send email from. Everything was going smoothly for me for 7 years until the entire Digital Ocean netblock my static IP was in landed in a permanent blacklist due to enough of the other IPs in that block having repeated complaints. I don't remember the mailing blacklist it was on but unblocking that single IP required the netblock owner (Digital Ocean) contacting the blacklist provider directly
this is why persons self-hosting email servers are much more likely to have success using a small to medium sized, trusted local ISP where you can establish a relationship with the persons who run the ASN. And determine for certain that the ipv4 /24 your mail server's /32 is contained within does not contain random other $5 to $30/month people buying VPS/VMs/low-budget-dedicated-servers with credit cards.
If you can have a high degree of confidence that no outgoing smtp spam traffic has ever been emitted from any of the other IPs adjacent to where you're hosted, the opaque blacklists of the big mail receiving providers (gmail, etc) are much less likely to consider your legit traffic as spam.
I think the vastly different experiences have a lot to do with the quality and scale of the ISP. Best results with small, good quality ISPs.
Also running my own servers for personal and business, and working well.
But when we tried to use one of the large VM providers the experience was much less reliable. Despite ensuring the IP was not on the various block lists etc. mails would be accepted and silently discarded by recipients ISPs, perhaps due to the level of abuse of these IP ranges.
Exactly. I’ve been running my own email server for over a decade not because I think I’m artisanal, but for practical reasons. But I don’t send out spam or “newletters”.
I've run a hobby website for about 15 years that does not even have a newsletter of any kind, and includes "stop sending me emails" in each transactional email (all users are double opt-in verified), and password resets are still not delivered half the time to gmail addresses.
In my experience people telling these stories often do send spam, but they don't believe they do. ("It's not spam, it's a Newsletter. No, it has no unsubscribe link. These are people that agreed to be put on the newsletter by clicking on some ToS they never read, and they can unsubscribe by some arcane mechanism that we will make as complicated as we can. But we're definitely not spammers.")
Yes. I do get that impression from most complainers.
I send from my own domains, and if I sent it, I wrote and addressed it personally.
I've run an email server for 10 years now and by and large this is the case. I am the only person that uses my domain/IP/mailserver. I know it doesn't send spam. I've still been blocked by MS Office 365, marked as spam by google, etc, every few years. It's quite a hassle to get unblocked involving lots of lying about having a Microsoft account or the like to tech support till you get to techs who actually know what a mailserver is.
Frankly, I'm shocked you've never been arbitrarily blocked and I find your insinuations offensive.
The last time I was getting blocked it was the solarwinds fiasco where their internal mail tunneling/forwarding and filtering setup broke all DKIM and suddenly solarwinds users like NOAA.gov were rejecting me and adding me to naughty lists. There was no fallout for the megacorps and their broken setups. There was only damage to independent mailserver operators doing the right thing.
I deal with this every day. Personal fully controlled server. I don’t conduct business over this server, have only one email, a personal email, associated with it.
I hear this all the time, but I question how true it is. I've been running my own mail servers for decades, and I've never had any problems with sending or receiving mail. I suspect anyone who properly configures their server will be fine.
> I hear this all the time, but I question how true it is. I've been running my own mail servers for decades, and I've never had any problems with sending or receiving mail. I suspect anyone who properly configures their server will be fine.
At work I ran email servers professionally and with good deliverability for years. My own email server was arguably longer lived than those at work, just much lower volumes. IP block was clean, DKIM, SPF, rDNS, etc. all setup correctly.
I thought I had no deliverability issues. I interacted with mailing lists regularly, the odd email to friends and family and I was firmly in your camp until I had to deal with a death in the family.
I think this was shortly after Microsoft BPOS became Office365. It became very very clear very very rapidly that to certain orgs I just wasn't hitting the inbox. And there was jack shit I could do about it. That was the end of my mail server, and it's certainly got worse over time.
The problem is delivery problems are almost undetectable.
If I send an email to a corporation’s customer support, or to a distant relation, or to an open source mailing list, and I don’t get a reply, it could be a delivery problem - or it could just be that they didn’t decide to reply.
For corporate support, that’s totally on them. Checking the spam folder is customer support. The “oh sorry your email went to spam!” is one of the few times I express dissatisfaction to a company. It’s not my responsibility to make sure my email doesn’t go to their spam folder. Not when I’ve taken all the right steps to make sure my emails are not marked as spam. If you have customers, you have to check your spam folder! It’s not foolproof.
You seem to not realize many email providers just drop emails (often after accepting it) instead of putting it in spam folders. So even your suggestion is of no help in that situation.
>just because they can and will do anything to maintain their monopoly.
This is a popular opinion on HN but it doesn't seem at all inline with reality. Email isn't exactly a real money maker for anyone. And the amount of email spam and abuse is immense. Filtering out most unknown providers is unfortunately extremely effective. Almost all spam wiped out with a simple check.
Maybe the ideal solution would be to let you link your custom email domain with a google account so you can have your google account vouch for the legitimacy of your custom domain. But even then, some of the time your email server actually is just blasting out spam without you knowing it.
Email itself maybe is not a money maker, but my company just went to 365 and 90% of the justification to management is "were switching email providers". Microsoft and Google's small business offerings are inextricably dependent on email first.
My own experience from running a private e-mail server the past 5-6 years is that the problem more than anything else is garbage "e-mail gateway" products, like e.g. Cyren GlobalView and Proofpoint, that gets in the way.
There's a tendency to perma-reject e-mail coming from "not seen before" domains despite the e-mail passing FCrDNS + SPF + DKIM + DMARC validation, which makes it difficult for private e-mail server users to get through to people.
This is exactly it. Email is now just another way to squeeze companies and private individuals alike instead of a cheap, secure and free way to communicate. Peer-to-peer email was worth having, in spite of the downsides.
I’ve seen a lot of small businesses go from $50 / year to $500+ / year. And from their point of view all they get is a bunch of nagging about 2FA and a much bigger target on their back when it comes to phishing.
And my retort is “that you know of”. If mine isn’t getting to you, who else’s isn’t? (For businesspeople) how much business are you losing because Google isn’t letting mail through? It’s one thing if the server is declining email and telling you why. It’s another thing to silently hide email.
Hmm, guess it's time for a counter-attack : "Sorry, but it's too much hassle to send e-mail to gmail/outlook, please use another provider if you want to communicate." ?
I post on a mailing list where one member has configured his server to reject all emails from Gmail. Inevitably we end up getting messages sent to the list which begin, "Direct emails to <guy> are being rejected, so I'm sending it via the list, sorry for the noise!"
The unspoken "you silly prick" gets louder every time this happens.
Every time my artisanal mails went to spam it was an overzealous corporate spam filter (mostly for mails with attachments). Never from one of the big hosts.
Having run my own email for the last two years, this is the number one problem I encounter. Somehow every official step published in terms of standards for securing email servers is not enough to appease large provider such that they’ll deliver your mail and not relegate it to spam.
The only problem I've had are with small players. You can't seem to reach anybody in charge of configuring and they do stupid shit that doesn't actually work.
The big players all have a process and followup within days.
Sending to Gmail/Workspace and O365 is by far the easiest case for me. It's the random enterprise email servers that don't like new gTLDs like '.xyz' who cause the most headache.
The suggestion that they will try muscle you out to maintain their monopoly is a bit alarmist. If you’re not sending spam, and your email infrastructure includes strong DMARC and SPF policies, then it’s unlikely that your reputation will be tarnished simply because it isn’t part of the Gmail or Exchange Online ecosystems.
I’d argue that the vast bulk of email is sent from dedicated providers like Sendgrid which are built on the same tech that might be found in any given ‘artisanal’ on-prem service.
Yeah I stopped running my own. I kept getting blocked by Microsoft in particular (mainly consumer recipients at live.com and outlook.com, strange enough not corporate O365 users!). I'm 100% sure I did not send any spam, the only emails going to those addresses were legit from a family member. DMARC and SPF were all set perfectly, relays blocked, I was not on any spamlist and I never have been either.
Literally every month I got blocked again because my server did not have enough reputation. Kept logging tickets to get it unblocked and then a month later it was back. One time I did manage to get a personal email back from a guy in India. Said that it was because my mailserver did not send enough legitimate mail for their algorithm to trust it.
So the lack of spam is not enough anymore to be blocked. You actually have to send a load of legit traffic to build up 'reputation'. Now just being a small time sender is a problem. This way the big players can just carve out a bigger market for themselves. They basically break the decentralised concept of email by doing this.
In the end I moved to O365, which felt bad because I didn't want to reward them for their behaviour. But we moved to it at work too and I wanted an instance with full admin rights to explore. My contract is up next year so I may change then if I can find a party that does it well and ideally cheaper.
> Said that it was because my mailserver did not send enough legitimate mail for their algorithm to trust it.
In other words, a small self-hosted email server will be considered a spammer until it starts sending out large amounts of email? Maybe that can be automated...
In theory it would be simple to provide a "cloudworkers cooperative" kind of service that just bundles the outbound mail so that traffic is sufficiently large to be whitelisted by the big providers. The two biggest problems are A. Scaling up sufficiently without attracting Spammers. Because even a single Spammer can ruin your reputation forever. So ideally you'd have a tight knit group of friends or similar. Even then you could hardly assure than no one ever gets hacked. B. Edge Cases. Even if your US or West European Traffic is sufficient to be whitelisted by all major Providers, how do you ensure that the occasional Email to a customer of an Indonesian ISP does not get blocked by their provider...
Yeah SMTP relaying is quite common. The problem is due to email architecture, to my knowledge, that same relay is going to be able to read your incoming emails because remote servers will block emails from user@endserver.org sent from relay.net unless endserver.org has MX entries pointing to relay.net.
'large amounts' is also pretty relative, I'm sure if you had a small team of 10-25 employees on a self-hosted mail server (preferably with a static IP via the ISP) you'd be taken seriously pretty quickly versus only you sending an email once a week or less.
I'm sorry this happened to you, but it's a shame. You end up giving money to your perpetrators and leaving the rest of us in the same situation you were in previously.
Maybe a hosting coop could be an option? Large enough for reputation but ethical enough to still federate with smaller hosts?
Agreed. But I did learn a lot from it. I needed that because in our large organisation at work the admin rights are highly compartmentalized. And this way I was able to understand what other admins were and weren't able to do.
You can actually get a free test tenant from MS for 3 months but setting up a real production environment is much better than doing some tests.
I tried office 365 for email this year but couldn’t get the marketing emails from Microsoft under control. No matter how much time I spent trawling through the settings menus. Almost every email I got was about some security update or promotion from some ms product I did not use and had no intention of using. And I was paying for O365 too!
It is possible. I managed to stop them in the end. One of the many admin sites if I recall correctly (seriously, they have an office admin portal, exchange online admin, Azure Ad portal and everything is spread out across those)
I host my own email and I have the same problem with MS. Perhaps this is something for the new Digital Markets Act and interoperability laws in the EU to handle.
You can configure postfix to relay emails to certain domains through a 3rd party SMTP service like SES. The MS domains give all of us the same problems, there is no other solution.
You may just have been unlucky with your IP block having spammers. How were you hosting it - own ISP or another provider?
I have not had deliverability issues for years with my Kimsufi (OVH France) server. While I am confident my server is well configured using best practices, I suspect some of it is also just luck not to be in the same IP block as a spammer.
I was using a colocation hoster in Belgium. They actually moved me to another netblock to test (they were a really nice small company). But the same happened.
I heard Kimsufi is indeed pretty bad as it's so cheap people tend to use it for 'throwaway' purposes. It's basically the white label budget brand of OVH :)
I've had 2 kimsufi email servers and both were fine. It shares the same data centres as OVH so I guess IP ranges are similar. No problems with blacklisting based on anything other than my own misconfiguration so far, and it's been maybe 8 years.
I had a very similar experience. Is there any cheaper option than just using aws SES nowadays ? (for outbound only! don't understand why people would pay the same rate for inbound)
My concern is what happens when aws decides to massively increase rates...
> This way the big players can just carve out a bigger market for themselves.
Or it’s because there is a near infinite number of domains so it’s relatively simple for spammers to avoid bad rep blocks by grabbing new domains and starting fresh.
Moved away from outlook.com hosting a while ago since so much legitimate transactional email went to spam whilst actual spam easily got through. Now, when outlook forwards to gmail, gmail catch it before it hits the inbox.
Except of course that it isn't an artisanal choice, a very practical one that is made increasingly impossible by the few very large email providers that are left. It should be as simple as hosting a web server.
Speaking of which, how long before it won't be possible to host your own web server?
On another note: the biggest source of spam is gmail itself, and guess what, that makes it to my inbox just fine, because what could possibly be wrong with someone using google as their source. Spam was annoying but it was never an actual problem. The consolidation of the internet into a handful of players is a problem.
> Speaking of which, how long before it won't be possible to host your own web server?
It's increasingly getting harder and harder. Recently I was trying to watch a TV show with my friends using a self-hosted Plex server, which was located in one of my friend's house, connected via a gigabit, albeit residential link. Another friend was using LTE internet at that time. He couldn't watch the show, because his connection was so slow, but when he did a speed test the download speed was good enough (100+ Mbit).
Turns out the mobile carrier was throttling connections to select IP ranges to about 1 Mbit (we tested that with a few other IPs). I reckon it was to cripple peer-to-peer protocols. So I guess it's a matter of time until you will be allowed only to connect to certain IP addresses owned by the biggest companies (AWS, Azure, GCP) and nothing else.
Net neutrality wouldn’t fix this is if the issue is a peering problem (which is very common today). The internet has become so centralized that ISPs cheap out on transit and just direct peer to all of the big content providers.
Speaking of which, how long before it won't be possible to host your own web server?
Maybe its just a matter of time for some. For me personally, I could not possibly care less if all the free mail providers blocked me some day. If something is important I can call people and tell them to go to https://mydomain.tld/theirName/ to grab files. I have used this method with non technical people including lawyers without issue. They prefer of course to use their own secure portals. I do acknowledge that running my own mail server may get more expensive with time as I may have to use providers that and more vigilant about keeping abusers off their network.
As for web servers why would I not be able to run my own servers? I can rent VM's, physical servers, racks, cages.
I am just speaking for myself but I will never give in to the bully anti-competitive behavior of the likes of Google and as for ISP's I will not use one that blocks ports or protocols. If there is any blocking to be done it must be done by me. I would never fund an ISP that uses CG-NAT or rate limits something by protocol or port. I realize some people have limited options but at least in terms of blocking and rate limiting, those ISP's are shooting themselves in the feet given that providers like Starlink and various 5G providers will be more common place soon.
"Except of course that it isn't an artisanal choice, a very practical one that is made increasingly impossible by the few very large email providers that are left. It should be as simple as hosting a web server."
I don't get this one. How do large email providers make it difficult to host your own email?
I host my own email. It was a pain to setup so I try not to touch it since it is running fine.
Setting up email on your own server is just complicated unless you install server management software. I am not sure big email providers are to blame for this.
> How do large email providers make it difficult to host your own email?
By randomly marking your email as spam without any recourse. This may be because they blacklist your provider en bloc, your IP address or some subnet, because they feel like it, it's Tuesday or because their spam filters suck.
But it happens and it happens often enough that running a business in that way will cost you money, sometimes lots of it.
"By randomly marking your email as spam without any recourse."
Correct.
I'd like to describe how badly this is implemented:
I run my own mail server and I have a 15+ year history of emailing (mywife)@gmail.com.
On a regular basis (mywife)@gmail.com will email me, and I will respond to her email and my response will go to her junk/spam folder.
And there is no alert, no bounce, no notification.
Let's unpack this:
Google (gmail) knows that these two email addresses converse back and forth, regularly, with a 15+ year history. Google knows that their own user initiated this conversation. Google knows my email is a response to their users email. Google knows my address has never been marked as spam/junk.
So, what kind of unimaginably bad heuristics would have to be in employ to allow this to happen ?
To be honest, this wouldn't bother me that much - I don't think google owes me anything and my wife doesn't pay for their service. What makes me so, so angry is that they behave this way without any notification or bounce email.
Business use of email tends to look a lot like spam and people mark it as such. An appointment reminder or notification that something just shipped is generally fine. Send out mass notification of your holiday sales and that’s going into someone’s spam folder.
Not so much a question of luck, sending out sipping notifications that for example include advertising is risky. Sending a high volume of appointment reminders for the same appointment is similarly problematic.
I don’t mean that’s the only way to trip up, there are a lot of unspoken self hosing email rules. Don’t use public data centers, don’t send news letters etc.
My email server is only used as a personal server for a few select friends and family. They absolutely do not send and have never sent anything that could remotely be considered spam. Everything in our setup is picture perfect (SPF, DKIM, DMARC, PTR records, etc). We still can't get email onto Microsoft's servers without it being marked as spam.
You are still hosting it fine. They just decide that you or your messages are suspect.
Also one thing - if people actually want your email they will contact you if they don't get an expected email. If they don't want your messages it is spam.
Even if your ip address/domain is not in the blacklist right now, it only takes a few people marking your correspondence as spam for it to be blacklisted. Since everyone is on these big free providers, nobody will ever see a single email from you any more. With less centrally controlled email, that would not be possible. I think that is the problem everyone is talking about.
I generally don't check my personal spam folder. I've honestly not seen any false positives with Fastmail. But I certainly do have to check every now and then for my work O365 account which is pretty bad at marking legitimate mail as spam. YMMV of course.
> How do large email providers make it difficult to host your own email?
By not delivering mail sent by your mailserver to mailboxes hosted by them. There's not much use for an own server, if your mail won't be received by most users on gmail or hotmail.
The problem with e-mail, and with other forms of communication, is that two parties (or their service providers) need to co-operate. You can run your own e-mail server just fine, but Google, Microsoft and friends might consider you to be a spammer or silently block your e-mail just because.
What if email was based on a whitelist instead of a blacklist? So you'd only receive email from addresses of people you've already established contact with some other way (maybe using conventional email)? This eliminates spam and if the big providers supported this, it could also enable them to stop blackholing innocent servers (though whether they care is another question).
You'll get it when Microsoft decides you are a spammer for no other reason then sending email from port 25 from your house. Or when you can't seem to sign up for a service... until you use your old Gmail address.
Yeah, there was (is?) a period of time where viruses were used to send spam so if you got infected you'd suddenly be sending out a lot of SMTP traffic from a residential IP address. The entire industry adopted the practice of not trusting residential ips. Then the spammers shifted to cheap VPS providers and ip and netblock black lists became more common.
[citation needed]; is this actually going out from gmail or does it just use gmail return addresses?
I too used to run my own email from about 2000-2010, but the maintenance overhead is quite stressful especially because it always happens for critical times or critical emails.
Almost all spam I receive is from Gmail. It's gotten so bad I've actually setup a filter that routes everything from @gmail.com into spam - except for some whitelisted email addresses. G Suite is fine, it's only @gmail.com that is an issue
And yes, it's genuinely from Gmail; valid SPF, valid DKIM, came from a Google IP address, etc...
To say the biggest source is Gmail might be technically wrong though - I suspect there's a large volume of spam that Migadu (my provider) is dropping before it even reaches my inbox, i.e. emails that it is 100% sure are spam and it can just drop. Nevertheless, an overwhelming amount of spam I observe/have to deal with is coming from Gmail. Second to that is outlook/hotmail.
> To say the biggest source is Gmail might be technically wrong though - I suspect there's a large volume of spam that Migadu (my provider) is dropping before it even reaches my inbox, i.e. emails that it is 100% sure are spam and it can just drop. Nevertheless, an overwhelming amount of spam I observe/have to deal with is coming from Gmail. Second to that is outlook/hotmail.
This. It's more likely to be survivorship bias -- the gmail emails happen to survive because gmail is more trusted.
The citation you are looking for is my inbox. That's the spam that still makes it through and there is quite a bit of it, conversely some ham consistently gets misclassified as spam or just simply disappears entirely.
I will echo this experience. An example of an email that made it through from a gmail.com account (abbreviated, it also contained links to some apps (the main purpose I assume) and much more text):
شهر مجاني عند الاشتراك السنوي $ الأسئلة الشائعة
In 1979, LA residents were wearing masks — because of smog Los Angeles Times staff photographer Boris Yaro photographed Sera Segal-Alsberg on Crescent Heights Boulevard in West Hollywood Segal-Alsberg, an artist-instructor, was en route to teach a class at the Los Angeles County Museum of Art
للمزيد من الأسئلة
— In another sign of live entertainment’s rebirth, Bruce Springsteen returned to Broadway over the weekend
يقوم الموظفين بتسجيل حضورهم ، انصرافهم الشركات العصرية مع الاستفادة القصوى من الإمكانيات الهائلة التي تقدمها لنا تكنولوجيا العصر أو الهاتف المتحرك ( الجوال ) أو إذا كنت تستخدم الحاسب فيمكنك استخدام أو يمكن للإداري تحميل هذا التطبيق على جهاز تابلت اشترك في النظام 10
Diverse yet divided cities
واحد أو عدة أجهزة ثم يضعها اشترك في النظام تسجيل دخول فقط في الأفرع المسموح له بالتبصيم فيها | أسبوعين كما أن الموظف لديك يستطيع التبصيم في ثوان قليلة بريد إلكتروني هو نظام إلكتروني قوي وحديث يستخدم لتسجيل فهل يمكن استخدام النظام في جميع هذه الأفرع أدخل بيانات الأفرع إن وجدت الموظفين
Experts say the Delta variant poses a greater chance of infection for unvaccinated people if they are exposed The variant, first identified in India, may be twice as transmissible as the conventional coronavirus strains It has been responsible for the rise in cases recently in India, the United Kingdom and elsewhere
في مداخل الشركة أو أفرعها المختلفة يمكنك الاعتماد على أي جهاز إلكتورني حديث أو حتى قديم في تسجيل ومتابعة تبصيمات الموظفين ، إعرف المزيد اشترك في النظام مجانا , أدخل بيانات الشركة والموظفين
When I ran my own mail server some years ago I was shocked at the amount of spam originating from Google. Definitely their IP addresses as I would routinely get other legitimite mail from the same IP ranges. Was quite a challenge dealing with these spam as it wasn't as simple as blocking their ip ranges as the vast majority of my personal contacts use Google. Never saw the same from Microsoft, Apple, etc.
There's quite a lot of small providers left and thriving - I've recently migrated from Gmail to mailbox.org, set up inbox encryption with my own key and can't be happier about it.
It's not as feature-rich as Gmail, and webmail with your own encryption key is not usable, but desktop (Thunderbird) and mobile (K9 mail) clients fully cover my use cases. Cheaper than Google Workspace, too.
Last year I was forced to migrate my 90 year old neighbour to webmail. We initially chose Gmail but ended up with outlook.office.com due to horrible IT policies at the University where she is emeritus.
Walk through a computer interface with a 90 year old sometime. It is eye opening. Both webmail systems were utter design hell.
The list of stuff that tripped my friend up is long. Two examples: Gmail has pencils everywhere and at least two different styles to compose a message (chat style, big screen compose, reply style too I think.) Microsoft’s product has a typeahead for the To: field that ignores your contacts list and instead uses the institutional one,
so typing “Anne” pulls up every Anne you’ve never heard of @youruni.com and not your friend Anne @gmail.com.
Gmail is also punitively fussy about receiving IPv6 mail but only on of its mail exchangers, so one in N mails get rejected. Great. Microsoft outlook requires you to scroll down each thread when opening it to see if “new message” meant the one at the top, or “new messages” plural further down.
You and I have become inured to this crap because we are comfortable solving problems with computers. For others, these products are very hard to use.
The one consistently brilliant client I use is the iOS mail app, via imap, to my personal mail host.
If I have business outcomes that are dependent on people receiving my emails, it's difficult for me to be in a situation where the other person may need to, at some point, be told that they aren't getting my emails because I'm on some kind of "hipster email system" which they are no doubt going to perceive it to be.
They're probably going to wonder why they are wasting their time with me when I can't even get something "as simple as email" right.
I see it the opposite way. I send a business email, it does not arrive, or it arrives after 20 minutes. When using the company's gmail, I cannot get any diagnostics - is it stuck somewhere in the queue? on their side, on our side? is it being greylisted? when will the server retry and the greylisting will allow the email to pass? is the target mailbox full? was there a TLS problem with the primary MX server and the backup one was used? When using my "hipster solution" (actually Postfix) I just grep their domain in mail.log and immediately see what has happened; I can use postqueue to retry or remove the email from the queue, etc.
AFAIK there is some interface in business gsuite, available only to domain admin and not regular users, where a limited subset of the above can be done.
The same applies for inbound email too -- they claim they are trying to send me an email, or getting an email from something like a network printer fails. Good luck trying to find inbound email logs for the entire gmail.
Well that's just because you're presenting it completely incorrectly.
It's their system that is misclassifying their email, not you being on a "hipster" email system, whatever that means. At no point is it your system's fault so don't even begin to start phrasing it as such.
Non-technical folks will just tell you that they don't have this problem with anyone else, and will likely get more annoyed when you present it that way. Since it's 2021, depending on which part of the world you are in, you may even be accused of "gaslighting".
Especially since some systems will just silently drop messages that seem "too spammy" instead of just sending them to the spam box.
From your perspective, your email was delivered successfully.
But to the recipient, you claim to have sent an email that is nowhere to be found in their inbox or spam folder, and if they contacted IT (keep in mind that 90% of people won't even bother doing this much), they'd say it was never received - why should they believe you?
It's amazing how quickly people with strong feelings about email come out when any mention of self hosting is made.
They can't prove a negative, but they valiantly try, often using the very points that are excellent reasons to self-host as reasons we should all just give up.
"It doesn't work for me, so it won't work for you" is rather weak when the reasons aren't articulated, when generalizations are made without detail, when hardly any attempt at all was made to find the root of the problem. Your failure is not the same as my quarter century of success.
They also tries to make everything all or nothing, which plainly lacks any imagination. I wouldn't want an admin on my staff who tries something, then gives up at the littlest of hurdles.
There are plenty of issues, but there are also plenty of solutions. You're having issues sending because your netblock sucks and you don't have money for something better? Pay a few dollars to smarthost, for instance.
With Google and Outlook, you get no determinism, no accessible logging, no clear view about their filtering rules, no assurance that your outgoing email won't get blocked, anyway, because of the tremendous amounts of spam these monopolies allow...
We should encourage MORE people to self-host email, not less. We should never just assume the monopolies are the best we can do.
> There are plenty of issues, but there are also plenty of solutions. You're having issues sending because your netblock sucks and you don't have money for something better? Pay a few dollars to smarthost, for instance.
I do and outlook.com still gives a shit and doesn't accept incoming emails. There is also no way to get their attention for this issue, as they simply don't reply to complaints about that. How to solve issues like that?
> We should encourage MORE people to self-host email, not less. We should never just assume the monopolies are the best we can do.
In my dealings with outlook postmaster my experience is as follows:
They ignore any argument you make or proof you show them that their servers are accepting emails and then silently dropping them. So that's basically the experience you mention.
However they will immediately unban your IP when you mention that you will recommend your customers switch away from Outlook email since it is unreliable.
That's for a totally clean IP with no spam issues.
How are you going to make that recommendation to your customers if they can't receive your emails?
(Of course I'm being facetious; other communication channels exist. But the idea made me think of the interrogation scene in The Matrix with Microsoft in the role of Agent Smith.)
What kind of mail are you sending? Are you getting anything on your abuse@ address? Are you getting any feedback on JMRP [0]? If all else fails, the mailop mailing list [1] can get you closer to someone on one of these networks to help.
>There is also no way to get their attention for this issue, as they simply don't reply to complaints about that. How to solve issues like that?
At one point, after getting one-too-many rejections for a particular recipient, I started sending the postmaster of the recipient’s service an email every time I needed to contact the recipient. That resolved the issue pretty quickly. You can always try annoying the postmaster.
Just personal emails. Volume would be somewhere around one email per month or so.
> Are you getting anything on your abuse@ address?
Nope.
> Are you getting any feedback on JMRP [0]?
That seems to require a Microsoft account and that's something I neither have nor want. I believe sending emails has to work without having to register an account for each provider you're interacting with.
> If all else fails, the mailop mailing list [1] can get you closer to someone on one of these networks to help.
Thanks for the tip, although I'm not sure if I'm the right audience for this list, as I just run a personal mail server with very low volume.
> If you want to send mail to recipients who have accounts at big email providers, be aware that all of the above cannot guarantee that these providers won’t reject your mail, put it straight into recipient’s spam folder or just silently discard it - they just impose their own rules on anyone and you virtually can’t do anything about it.
My guess is, that the reason for the problems is the same as quoted in another comment (https://news.ycombinator.com/item?id=29673347), stating that the mail volume is just too low for outlook.com to establish enough trust in the sending mail server.
If you look around on the internet, there are plenty of other people with the same issues with outlook.com. To me it seems Microsoft is doing something fundamentally hostile to small mail servers there. Interestingly enough sending to Office 365 hosted email addresses works just fine.
>That seems to require a Microsoft account and that's something I neither have nor want. I believe sending emails has to work without having to register an account for each provider you're interacting with.
Yes it does require signing up with them. I see that you’re taking a moral stance on this so I guess the best action is to just ask people to not use Microsoft email products, which is perfectly reasonable in my opinion.
For what it’s worth, I do have an account with them and I am very small scale and don’t have any deliverability issues with Microsoft.
>"It doesn't work for me, so it won't work for you" is rather weak
Maybe I missed it but I scrolled through the comments and I don't see anyone in this thread claiming the above. In fact, I see the people who had problems with personal mail server deliverability acknowledge that others may have success and "that's good for them."
unfortunately as long as you’re sending email to other people you’re subject to Google’s filtering rules, so it may be easier to work with a provider that operates at a scale google cares about… it’s all a racket if you ask me… their algo can doom any individual self-hoster to perpetual spam hell and they’ve got zero service
I've said elsewhere in the thread, but everyone has a circle of influence. Big Mail needs to maintain a reputation of being reliable. I can and have convinced people that e.g. Yahoo mail is not reliable for receiving mail and to stop using it. And those people will do the same with their people. I think that's why it's important for people to keep at it with these "artisanal mail servers" (with the help of mail packages that do automatic configuration). More people can pressure Big Mail and demand that they let us deliver our (legit) mail as smoothly and easily as possible.
The issue is the spam still gets through on every single one of these providers, so that reputation isn’t nearly as important. Reliably receiving emails is. If that aspect of their reputation is damaged, it may never recover.
Do you mean pay a few dollars to use someone else's email infrastructure? Or do you mean pay a few dollars more to set up a dedicated mail host in a better neighborhood?
Yes. I want to create a mail server that auto-rejects e-mail with externally hosted tracking pixels and returns a Mailer-Daemon error that tells the sender to re-send without malware.
Of course with Gmail I disable automatic image loading to avoid being tracked, but I want to take a more active stance against the practice.
I would just block all images loaded from external URLs, since any of them could be used for tracking, but the newsletters are less my concern, it's more the individuals that I would be concerned about, and they tend to be a almost-fully plain text e-mail with a 1x1 image from one of many well-known domains for tracking.
Emails can have "inline" attachments, where an image can be attached in the body of the email itself. So the image shows up without any network access, but it's generally base-64 encoded (costing 4x space) and ends up using a lot of email storage space.
Ohh, I was thinking 256/64 = 4x, but it's apparently it's 8 bits / 6 bits = 1.3333x. Thanks. Plus for email a little extra for a =\r\n on every line, so more like 1.4x.
If you do this you might not be able to go on a "paperless" billing plan with your bank or insurance company. I disable images too but financial institutions interpret "no tracking image requests" as "client's email address doesn't work" and they start mailing me paper again. Took me years to figure out the reason for this brain-dead lossage.
That's their fault then, I'll start returning their postal mail as undeliverable and if they give me further troubles I'll find another institution to do business with.
Yup, between the message trace and just reading the headers I have not been wanting for visibility... and as the Exchange Online admin I've found that 99% of deliverability issues end up resting with the outside party failing to read their own infra's NDRs anyways.
> … general security will not be as good as they have.
> Entirely "on premise" email is now an inferior thing for almost everyone.
I disagree on this one. Placing your email with a big player means that by definition, they have access to your mailbox (with sensitive stuff hopefully encrypted). To allow that you have to trust the big player and the countries where they reside.
They can drop you any time for political reasons, for dealing with a country that is considered an enemy of the host country of your provider. They may sell out your data.
You may still choose a big player, but understand how screwed you are.
Once you’ve sent or received email containing “sensitive stuff” you no longer control your data. Folks need to come to grips with the fact that email security is dead and hosting it yourself doesn’t fix this.
The author of the post should have included that in his post.
"Folks need to come to grips with the fact that email security is dead and hosting it yourself doesn’t fix this."
Sort of, but not entirely true ...
If you run your own mailserver then users of that mailserver can send and receive mail, to each other, without traversing a network. The mail never goes out on the net. That can be valuable/interesting.
This is true in both the webmail use-case (the text goes to the browser, ephemerally, encrypted with SSL) or the terminal/console (alpine) use-case (the text goes to the terminal, ephemerally, encrypted with SSH).
There's a certain cloud storage provider I know of whose internal / intra-company emails have never traversed the Internet ...
One thing running your own SMTP/IMAP protects you from is faceless lockouts. My partner was locked out of her Hotmail account and no matter how much evidence she provided Microsoft's shame-faced automated system refused to give her access. No court of appeal, nothing. Turns out she has an important email regarding her Indefinite Leave to Remain in the account containing a UAN she now needs access to in order to apply for a Biometric Residence Card to ensure she can continue to work in the UK. Fuck big email providers, I say. Legislation needs to be passed to make them accountable when locking-out users from their accounts.
Since from your comment she appears to be in the UK, GDPR right of access still applies. She can get a copy of the data, even if not access to the account.
Your comment raises an interesting point: If I, a non-EU citizen who doesn't live in the EU, get locked out of my gmail account, would traveling to the EU and invoking GDPR be a viable way for me to force google to hand over my emails?
If yes, then I sense business potential. I for one would be willing to pay a lot of money to access my inbox.
I recall reading that it's 6 days of living in the EU until it "kicks in." I don't have a source or that and can't seem to find one, so take it with a pinch of salt.
The article makes a string of assertions without evidence. For example, that an “artisinal” email server will not “measure up in usability, features, and performance to the email infrastructure that is run by big providers.”
One reason that I continue to run my own email server after more than a decade of trouble-free operation (thanks Postfix and Dovecot) is that it performs better than my Gmail account, which I maintain as a 3rd-level backup and for some email lists. Delivery and receipt from my personal email is so fast that I can use it for real-time conversations with anyone else on a good-performing server, almost like chatting on WhatApp. This doesn’t work with Gmail, because it takes so long for a message to leave their servers. Plus on my own system I don’t get the spam false positives that plague Gmail.
Those are only two ways that a personal server outperforms Gmail. I can do plenty of other things with it, because I control it, that are impossible with Gmail.
> The article makes a string of assertions without evidence.
Agreed, but the same observation could be made about most blog posts. The author is expressing his opinions based on experience and judgment. This particular author has a good reputation so HN readers take his opinions seriously.
Fair enough. But in this case his opinions are incorrect and arbitrary.
The reason I and many others maintain our own email servers is not to be “cute” (a word used in the article) but because we want superior usability, features, and performance over what the major providers can offer. Otherwise, why would I bother? The only issue is that some people claim to experience delivery problems. Others, such as myself, say that delivery is at least as good as from major providers.
And receipt is far better than when using a big provider. Unlike the unfortunate users of hotmail, I actually get all my email.
EDIT: And not only does (for example) Gmail run a poorly performing email service, but their web client for interacting with email is broken: https://lwn.net/Articles/837960/
I still run my own email system (postfix/dovecot for imap), mostly for one reason: the virtual username function of postfix:
I configured postfix with:
recipient_delimiter = .
which gives me unlimited dynamic virtual addresses (username.<something>@mydomain), so I know where spam/leaks come from if I get unsolicited mail directed to `username.<unique_name_per_registration>`, and it makes it trivial to block.
I know that you can do the same thing with google addresses using + as a delimited, but the + sign is often not allowed in dumb email checks. Also spammers probably know about + and strip it automatically anyway...
If you're running your own server, you can just setup catch-all account and use something like $(printf %s news.ycombinator.com | sha256sum | head -c 12)@mymail.com for further privacy.
You don't need to run your own email server for this. I do this with a catch-all in Fastmail for $50/year or something. I'm pretty sure Gmail and most others can do this too.
I've been doing something with dash as the recipient delimiter since the late 1990s and it's been great. But that became a pain when I wanted to switch to hosted email, as many providers wouldn't support it.
I eventually ended up at Fastmail, as they let you build custom Sieve scripts that can do this kind of remapping without having to run your own mail server.
Love that, too! I've always been amazed that spammers aren't able (to my knowledge) to defeat such a simple scheme by removing the . or + in the local part.
I believe you can insert dots as you wish but not use it like the plus sign. So abc@gmail.com is the same as a.b.c@gmail.com but abc.new@gmail.com is another account than abc@gmail.com.
It feels weirdly like people saying "this is all too complicated, only experts can handle it" which is giving up a lot of agency. Almost like "Big Tech" is shadow writing pieces to discourage you from trying to escape their grasp :-).
Running your own email server and domain, for people who like systems, is fun. Just like people who do their own oil changes and car maintenance, or people who build their own furniture, etc.
Interestingly enough, this suggests there are some startup opportunities for folks who want to make this stuff a bit easier. Three things I think would be interesting side projects would be 'spam killing' (Barracuda does this as a service for Enterprise, I bet you could do it in clever ways for individuals), "post office" which is a known good relay server with mail agents that you can forward your mail through (think Lets Encrypt but for mail delivery), and a remote access client for phones. Alternatively an AWS offering of a packaged mail server (think WPEngine but for mail) has possibilities as well.
I've been running my own email since i was 14 years old, back then it was hosted on my home ADSL connection. Now I'm on fiber and still running my own email setup, but the end is near..
For the reason not mentioned in the article, it's getting increasingly harder to actually get a public routable static IP address and also be allowed to either send traffic on port 25, or use the ISPs relay host to actually send the email.
There's a few ways to approach this question. One is to mention community networks (DIY ISPs) which will ensure you always have a public IP without filtering. Some even provide VPN access so that you can use your filtered internet to acquire a publicly-routable IP. This is a common pattern in the ffdn.org federation of non-profit ISPs.
Another one is to mention hosting coops (libreho.st/chatons.org) and how they could be employed in limited-network situations. On the web, we have SNI/eSNI-aware proxying which enables multiple servers to share a single IP without revealing their private keys to the reverse proxy. I don't know of an equivalent in the email world (because it's assumed there is only one MX with a canonical domain/DKIM per IP), but i'm all ears if you have suggestions!
Of course, we could mention onionMX and other key-routing systems (CJDNS..) but the problem is you need it to be supported on the other side as well, which is highly unlikely.
Some ISPs provide VPNs on their AS at cheapish price. For instant milkywan in France provide one with public constant IPv4 for 5€/month in France.
It obviously make the whole setup much more complicated, because it leads to a kinda multi-homing setup, but I think it's still reasonable.
This defeats the entire point of selfhosting whether it's for security concerns or autonomy. The ecological impact of VPS hosting every single service you need is also not negligible: datacenters require huge amounts of resources and infrastructure which a simple second-hand machine at home doesn't.
(also worth mentioning: email protocols were explicitly conceived so that uptime is not a worry)
Datacenters use energy and computer resources more efficiently than a machine at home, unless the machine at home is already running some other tasks that you can't move to the datacenter. A computer that's 99% idle is wasting most of the energy it consumes.
> Datacenters use energy and computer resources more efficiently
That is both true and misleading. Once the datacenter and all surrounding infrastructure (optic fibers, fuel pit, dedicated electricity lines, cooling equipment) and all server/networking hardware has been built, then you start having a better efficiency. If the whole cycle is taken into account, there's no way VPS can be as "green" as selfhosting.
A computer will usually take more energy to build than it will consume over its entire lifetime, so repurposing an existing machine is a good way to go (if you consider minerals-related pollution, even more so).
Also, when you're in a datacenter, servers will be changed every few years. For something as simple/lightweight as email, a 20y old computer will do just fine. A datacenter will renew its entire hardware a few times in that timeframe.
> A computer that's 99% idle is wasting most of the energy it consumes
That is true whether it's in a datacenter or at home. But of course you can share/mutualize resources with other people in order to mitigate this.
How often do you need to send on port 25? In my experience also running my own mail server, never. Receive from servers that don't support encryption yes, but never send. I always send using TLS and since maybe 5 years I've not had an issue with a receiving server not supporting it.
If you want to deliver email to other domains then you need to connect to port 25 on the destination domain MX server. As far as I know, best practice for the other ports (465, 587) is to require authentication and to reject anonymous submissions.
Port 25 is only required if the destination doesn't support TLS, I think. I've not opened port 25 outgoing on my server and I've had zero issues delivering sent mail to other servers for maybe 5 years.
I think OP might have meant "receiving on port 25 is getting difficult" rather than sending. The spec requires servers to support unencrypted deliveries over port 25, even though almost all servers use TLS these days.
Even with TLS, that is usually handled by issuing STARTTLS on TCP port 25. I can't find anything in the RFCs mentioning server-to-server smtp delivery happening on anything but port 25? Do you have a reference for that? In fact, even the MX for google domains (aspmx.l.google.com) does not listen on TCP port 465 or 587, only 25.
I'm not certain, so you might be more informed than me. It's possible my server is sending on port 25 and since the firewall I use doesn't block outgoing connection I just didn't notice. TIL!
It’s not so bad to run your own email server. Im doing it, because email is very sensitive data, and I don’t want to share it with Google or Microsoft or any other cloud provider.
The worst part is, that you can’t use any fancy email clients, because they all use proprietary protocols and once again cloud services. But imap and activesync works well enough too.
The most important thing is, to get a clean IP address. Don’t ever try to host your Mailserver on digital ocean for example, their IPs have such a bad reputation that some providers even block them on network level. Their whole subnets can’t connect to them, no possibility to get unblocked at all.
Once you found a legitimate hoster, check that your IP is not bkacklisted at any major provider. If it is, try to get other IPs, until you get a clean one. Don’t try to go through unblock-processes, that often won’t work.
And then you need to set up your server well. SPF, DKIM, DMARC, …
I've had real trouble with digital ocean as they just don't want to get involved in helping email providers at all. They don't want to do anything with managing IP reputation.
I know. I think there is a study, that showed digital ocean is the provider that sends the most spam emails, and just doesn’t care. As self defense, some providers just block their ips because of that. And if you try to unblock such an ip, they require you to do a lot of erfrort. You can skip it, if you get an IP from a provider with good reputation. AWS for example doesn’t kid around if you send spam. You need a special permission in the first place to even be able to use port 25 outgoing.
> (with DMARC signatures and other modern email practices)
DMARC does not provide signatures, DKIM does.
DMARC adds the DKIM 'alignment' requirement. Meaning that not just any DKIM signature will do, the public key (the DKIM DNS record) must be published under the administrative domain (the part after the '@' in the sender address).
DMARC also mandates SPF alignment (not that your should rely on SPF), meaning that the rfc5321.MailFrom and rfc5322.From address should be from the same administrative domain for the SPF to pass DMARC.
When either SPF or DKIM is aligned, you have a DMARC pass. Because SPF breaks with forwarding services, you shouldn't rely on it. DKIM + DMARC is the way to go.
Also funny that the author calls DMARC 'modern practice', since DMARC was introduced in March 2015, almost 7 years ago.
Email is honestly not that hard to get right, but you do have to get a lot of stuff configured correctly for everything to work well. I've never had a lot of problems with blacklisting and at one point, was running about 48,000 messages per day sending marketing emails (ecommerce sites, to subscribed users) and transactional emails. Deliverability issues were few and far between, and most often were caused by MS Exchange admins at distributors (where we would send orders via email) who just didn't understand how the internet worked (like blocking all domains that weren't .com, .net or .edu). My favorite was one who blocked .io because he personally only got spam from that TLD.
On the whole, just use a service makes sense in that the time you spend on email probably could be spent on things that make money. If you have an app that sends a lot of emails, understanding how it all works can be a very useful skill, too. So is learning to work with admins at big service providers.
The wild west days of email, with bang paths and "store and pray" delivery systems, those were fun. By the time `sendmail.cf` hacking was no longer a necessary skill, email had become industrialized. Today, why would you even want to try routing internet email through an RBBS net to WWIV net to some hackers custom Amiga board?
SMTP was the Ford Model T of electronic messaging. It slaughtered the previous visions of what the field needed to be. We can look back fondly at the older ideas and even re-implement their insights now, but the lessons of the market are written in big bold letters now.
I ran an email forwarding service https://mailwip.com (former hanami.run) that also support SMTP and IMAP.
IMHO, gmail is the best company out there that can accept your emails. The most common issue with gmail is email land in spam but they learn quickly and very rare just outrage reject IP. as in, gmail won't relying solely on reputation of IP but based o sender domain or so.
Compare with hotmail or icloud or some random email hosting service, they will reject your IP just outright.
I would say it's definetely tough and stressful to run an email services for everyone, but if you run it for yourself only(so you know you're good and won't send out random spam), I will say it ins't that bad.
My email is still the free/legacy version of Gsuite, from before it was called Gsuite and it was just a way to have your own domain but serviced by Gmail.
It's kind of a pain, because it's both not really a Google account, but also not a real Gsuite setup, and Google keeps quietly removing features from the legacy version of Gsuite. (You used to be able to use an external SMTP server to send as a different domain, but Google removed the UI to configure that in the free version)
I've been really hesitant to upgrade to modern Gsuite, because I'm worried about upgrading and having something go wrong in the process - Google support doesn't have the greatest reputation.
Much more needs to be said about the extreme advantages of paying someone vs. using a free service.
I've said this for over two decades: If you have a business, or even just a lot of important stuff going on in email (which is like everybody); it strikes me as insanity to not pay for the peace of mind that comes with "a human you can call up and say 'hey, why can't I get into my email' or 'hey, fix this please"
Versus what SO MANY PEOPLE use, which is "It's possible that your email will be removed from you entirely and you will have recourse because no one has a contract with you to fix it."
Also, a lot of software in the field has never made sense to me. I know what the parts do, but I couldn't tell you how to assemble it all. It all seems very old and seperate software more for historical reasons than anything else.
I ran mailinabox for a year or two, but eventually I just didn't want to maintain a piece of software I didn't understand where the documentation seemed actively hostile and presumptuous about me having read all the other parts. I'm sure the postfix docs make an okay reference, but understanding it as a whole, god no. I'd rather do Kubernetes from scratch.
Postfix and Dovecot are classic magic word projects - completely useless unless you can work out the magic words, and then they work fine.
The docs for magic word projects never to seem to prioritise essentials. So [obscure feature someone last used in 1984] gets equal billing with [essential fundamentals] and you have no idea which is which because - you haven't understood the docs yet.
I'm still running my own servers. I sorted out the spam issues, and they're basically zero maintenance now. But it certainly took a while, and a fair amount of copying other people's ideas of what a config file should look like, with plenty of trial and error.
Agree regarding learning the magic words. There''s probably no substitute to just diving in and setting up a server with a domain you can afford to get blacklisted for a few weeks while you make mistakes. But once you know the magic words, Dovecot's documentation is actually fairly decent these days: https://doc.dovecot.org/.
> I ran mailinabox for a year or two ... the documentation seemed actively hostile
Maintainer of Mail-in-a-Box here. I'm sorry you had that experience. Definitely was not the intention of the project to be hostile (but I can see how it might come off that way).
Hi Josh. I should've specified I meant the postfix documentation here. MiaB was wonderful, save for maybe skipping one Ubuntu LTS and leaving not quite a lot of time to migrate.
I ran my email server (several domains, a dozen or so accounts) for over a decade. This year I caved in and switched to hosted (gmail and fastmail).
For most of the time it's been smooth running, but I did have to do maintenance on the server every year or so, just in time to forget the intricacies and having to relearn them again.
Yeah, a few hours a year on that is not much. But there are many such small "auxiliary" things/chores and it adds.
There's so much things I would and could want do myself, and nowhere near enough time to do them. I have to pick my battles.
And figuring out how to fix sender rewrite to enable mail forwarding with SPF without accidentally allowing spam is not very high on my list of important things in life.
I run my own email server and it's not so bad. gmail is obviously the most troublesome "peer" - if it wasn't for the fact I need to communicate with some gmail users regularly then I'd just cut gmail off. You have to set some headers and DNS stuff, having a static IP helps, and obviously not allowing that IP to send spam too. Largely it works fine.
I do too, gmail is not a problem at all, but I have had the same static IP on AWS for about a decade, reverse DNS is mandatory for deliverability. My problem with Gmail is it is difficult to filter SPAM coming from their servers.
My biggest problem with mail delivery is sending mail to Microsoft properties. I've had to resort to sending those messages via SES.
Open source webmail solutions suck, so now I'm paying fastmail and forwarding incoming messages there.
Also run my one server for years without too much trouble. Hotmail is the one giving me the most hassle - for some reason they periodically block my IP address for apparently belonging to an ASN they block … except it belongs to an entirely different ASN. My provider (RamNode) say they’ve been trying to get someone there for years to fix their system but didn’t help, so I just fill in their unblock form now and again.
If you have any neighbourgh on your block that has sent spam, then the entire IP block gets a bad reputation. I moved away from OVH because of this. I no longer get the issue )on Vultr) but still get 100% blocked by outlook.com and blocked by Gmail if I'm the one initiating the conversation (I can mail people once they've emailed me at least once)
Their email referenced AS3150 ... Here is what the (clearly frustrated) RamNode support person had to say last time I enquired:
AS3150 is NTT, a large backbone provider: https://bgp.he.net /AS3150
RamNode runs AS3842 and AS198203. We have contacted them before - they don't know why nor care why their system is raising issue with AS3150 in regard to emails from our network.
But this issue isn't exclusive to our network either, and no other major email provider blocks emails like this.
A couple years so I set up a mailinabox server on a vm for my personal email. Incredibly simple, $5/month and “just works”; I can even sync my contacts and calendar.
My only complaint was the graylisting but I quickly resolved that with a configuration file.
I do away with the middle ground. I rent email services from businesses that specialize in email hosting and I own domain name. No way I will have my email serviced by Google / MS / Apple and the likes.
Who do you use? After seeing all the horror stories posted on HN, I have been worried about getting banned from $MegaCorp randomly and would like to minimize the blast radius if that were to happen.
I run my own email. But I think it is a software problem.
It would be nice of you could just install an email program that will set all the right settings for you. DNS, database, roles and rights, certificates, firewall and so on.
There is server management software that can do this but then you have the same problem: it is just complicated for most people.
As someone who run my own email server, I obviously disagree. Three things come to my mind:
First, SPAM filter is way overrated. I have next to zero SPAM filter, and am doing just fine. Yes, I got lots of SPAMs, but the volume of real SPAMs is dwarfed by the volume of ads that would pass through SPAM filters anyway, so why bother.
Lastly, the biggest pain I have is sending email to big providers such as gmail. I have everything setup correctly, DMARC, SPF, you name it. And my server is not on any block list that I can find, and yet they put my emails in the SPAM folder from time to time. In the name of fighting SPAM, they are sabotaging the original internet experience for everyone.
The comments all seem to be from people who think making an artisanal choice is a bad thing. There should be more art, experimentation, and expression in computing. If we've outgrown running our own mail servers as a practical choice, because there are now more good options, and we can enjoy running a mail server as a more humanly choice, that's a good thing.
Not much has actually changed about the complexity of running a mail server in the last 20 years --- if anything it's gotten easier. What's changed is there are other, polished, turn-key options now. Great. (Those options tend to have spam policies that aren't friendly to the independent servers, but that's life.)
I don't think artisanal is necessarily "bad", but we should all acknowledge that it will be more work for a result that's maybe better but probably worse for most people.
I ran my own mail server for 20+ years, finally giving up a couple years ago. I strongly disagree that it has gotten easier. As the article makes clear, it's a much more complicated world. Things that have happened in the last 20 years include SPF, DKIM, DMARC, and the rise of providers like GMail. And if you really care about owning your bits, in some ways colocating hardware has gotten harder now that VMs are hugely dominant.
The new requirements to be a good mail server are significant work to understand and implement. The feedback loops are also poor: it's hard to know whether you really have them right.
But the real killer for me was opaque major providers like Google. Occasionally, they decided they didn't like my little mail server. I and a number of other sysadmins couldn't find anything wrong with my setup. But mail wouldn't arrive. I even had SRE friends inside Google and they couldn't find out anything; apparently the GMail folks are very secretive.
There are only so many missed business opportunities and disrupted personal relationships I was willing to put up with for my personal taste for running my own servers. Eventually I hit that limit and switched everything over to Fastmail. For me personally, it was a great decision. It's cheaper and more reliable, and never again will I have to get up in the middle of the night to go to a colo. In contrast to my spending a few hours here and there, they have a whole full-time staff sweating deliverability. It's great!
If people think running a mail server is fun, I say go for it. But even there I'd strongly urge them to consider whether "this looks fun" is the right spirit to bring to anything important to their lives, and whether it will stay fun when it breaks at the least convenient time. So maybe keep it fun by using it only for things that don't really matter to you.
> Things that have happened in the last 20 years include SPF, DKIM, DMARC
Right, of course. The protocols are more complex. (Add TLS, MTA-STS...) But whereas 20 years ago you _had_ to start from scratch and understand the whole stack, today that's just not necessary. There are numerous projects that make running a mail server readily possible without knowing e.g. the sendmail configuration macro language. And there are many many more good resources to learn it all if you want to know than there were 20 years ago. It is both a more complex technology and also undeniably easier for people to actually do it.
I wrote the email chapter for the book "Internet Secrets" in 2000, and I ran my own mail server 1997-2019, so I have a pretty good sense of what the landscape was like then versus now. QMail and Postfix were both out before 2001, so you didn't need to know sendmail at the time. You just had to be able to configure the mailer to get things up and running. And given that there were decent Linux distributions available, the technical challenge wasn't high.
The difference now is that from there, there's a lot more to understand if you actually want your email to get anywhere reliably. It's complicated, subtle, and much harder to resolve problems when you get it wrong. At the time, the biggest problem was bounces. Now deliverability has become a dark art.
Just out of curiosity, when did you start running your own mail server?
I think around 1998 or 1999. I don't think we disagree on the facts: I totally agree that there is a lot more to understand and that deliverability is a nightmare. No question. What I see is that today people can achieve a reasonable mail server while being an expert at less.
Hey! Thank you so much for MIAB. I've been using it for personal and business email since May of 2016. It has been an absolute joy to use and administer. I am truly grateful.
I've got a personal email server and an old gmail address from gmail-beta days.
I've never used gmail itself (that model doesn't fit my mind), but O do use that Google account for some minor stuff. Unfortunazely, I've repeatedly gotten email targetted at someone else having same first name initial and same last name on gmail (address is in the form of FLastname@gmail.com). I've usually been able to get through to those people to stop them and to get them to reach their targets, but in the last 24 months, a lady from Michigan is repeatedly giving out my email address for everything (I've got covid appts, doctor appts, movie tickets, responses to home buying inquires...). I have no idea how to stop this: this would've never happened with any provider that's not owning like 60% of the market.
I am constantly annoyed and I've considered both stopping mail forwarding from this account to mine (but then I might miss that YT premium notification) and I've tried reaching out to many humans on the other side. But I've so far resisted the urge to cancel those movie tickets or vaccination appts, but things just keep coming in.
I can't imagine how are people not overwhelmed by wrongly targetted email: there's more of it than spam I get on my personal server, so spam filtering would definitely not move me towards gmail. And actual spam also gets through on gmail!
I have a very short gmail address from the beta days and get probably 5-10 emails a day addressed to accounts that absolutely aren’t mine, Eg my username then a dot and another word, or my username spelled differently. I don’t know why google thought fuzzy-matching emails was a smart idea, but it really isn’t. I’ve gotten a lot of very private information and direct login links to a lot of stuff. I also get countless people putting my email address as theirs, to the point where I’ve given up trying to fix it and just delete it instantly.
To even use that gmail address I need to basically whitelist senders and filter them into folders and ignore the inbox completely.
Gmail, like so many of Googles services these days, is an absolute mess. Features no one asked for, blatant spam that gets through their checks while your actual emails go to the spam folder, and a constantly degrading UI that seems to be an experiment in how much you can annoy the user.
What you've described is common among many first adopters. I know someone who has a common first name and last name combo, signed up for gmail in mid-2000s and use the firstname+lastname@gmail.com.
He got many "weird" messages over the years from messages addressed to a religious minister, to a pro-gun individual and to NSFW account that he didn't sign up for.
I also get an endless amount of email for a few different people with the same name as me at my Gmail from 2005. I used to try to deal with it but now it's just amusing, especially the photos, family chain emails etc. My name alike is Canadian so it's a little glimpse across the border
I don’t think you can stop it. I have to imagine that this woman doesn’t have an email address and just gives out what she thought was a fake one, since so many things require it. Otherwise, how could she not notice she never receives anything?
> Otherwise, how could she not notice she never receives anything?
She may be getting it at f.lastname@gmail.com.
I signed up for Gmail the first or second day it came out with first.last@gmail.com, but hardly ever log(ged) in over the years. A little while ago I did go into it and noticed getting a bunch of message to firstlast@gmail.com.
Now Gmail allows for 'customized' addresses in that you can drop a period anywhere and it will still go to your address. But this raised the risk of one person signing up with first.last and another person signing up with firstlast. Supposedly this is prevented, but I think that they did not catch this situation in the early days of the service, and so a bunch of OG accounts have cross-contamination.
Someone signed up for an Amex card using my wife's gmail address. This was years ago, and she still gets the Amex emails regularly (like, between daily and weekly). It's incredible both that the Amex customer doesn't notice, and also that Amex has absolutely no method whatsoever for a non-customer to contact them and fix the problem.
>Unfortunazely, I've repeatedly gotten email targetted at someone else having same first name initial and same last name on gmail (address is in the form of FLastname@gmail.com).
That's a user problem, not a technical one. (there is, of course, an XKCD for that).
I've got a popular initials/surname combo and I have a number of doppelgängers giving it out. The one I feel most sorry for is the trumpist and his scary NRA/pro-gun mail. It's really fierce stuff, I'm glad it's going into my spam folder rather than in front of a real human!
Note that the original post is from a purveyor of commercial email, he probably feels the heat from those artisanal servers.
You can run your own email servers(s) as long as you know what you are doing. Setting up your own is not for the faint of heart. However, cPanel will do the initial setup work. CSF will see to it that your mail server is moderately secure, and that the bots trying to kick down its doors are get banned. Setup of DMARC, SPF, etc increasingly gets automated. Mxtoolbox makes sure that all aspects of DNS and mailservers are correct. Finally, you can send mail to something like analyze.email, and they will score your server. If you score an 8 or better, your email will definitely go through (analyze.email will hand out demerits for NOT having a link in your email, for from and reply being the same, or for a lack of a list-unsubscribe.)
If you run your own business, an own website and email address is important for branding. Gmail sent “on behalf of” does not install a lot of trust.
Here are the downsides to outsourcing your email:
- Who will read your mail sitting on their server?
- Who will give your mail sitting on their server to anyone waving some legal papers?
- Who can kick you off their server without recourse, killing many years of investment into your email identity?
Here are the downsides to rolling your own email server:
- If an evil hacker invades your badly protected server, and uses it for gadzillions of spam, a nicer provider will turn off your outgoing email, a nastier one will null-route you no questions asked. That’s why you should start your career as an artisanal mailhoster on a VPS -- you can always rent another one. It gets nastier when a homelabber decides to do it from home. Hacker invades, Internet provider cancels your account for violation of TOS. You usually don’t have much choice when it comes to a new hardline provider, and it will take a while.
- A big danger are the countless blacklists. It is very hard not to be on any. Some will blacklist whole net blocks if one IP in the block misbehaves. Some are plainly extortionist; they want money for removal. Frankly, they should be taken to court.
A lot of the comments are talking about people running their own personal email server. However, the post is more about arguing that organizations running their email infrastructure will become rare, and I'm not sure I agree with the author's thesis.
The challenges there are rather different. Spam is less of a problem: in B2B the correspondent is well-known and will be whitelisted quickly, while one can pay for extra spam filtering if needed. User experience isn't a biggie because frankly the biggest users aren't the ones paying. The real issue is TCO vs data independence - and in my limited experience I still see plenty of organization which still run Exchange (or equivalent) on their own domains.
Been running my own mail server out of my house for about a decade. It has been glorious. I have always paid for 3rd party spam filtering, both inbound and outbound. Inbound and outbound through a 3rd party helps reduce the attack surface at my house.
My users do not need webmail and I do not offer it. Calendars are done in local clients and shared through invites.
I went with DC powered equipment, with battery backup on the input. My internet and email can stand at least a 24 hour power outage. Its also all solid state and no fans, no moving parts. Its been very reliable, though I do replace the batteries every couple of years. I'm probably over due for hardware updates at this point.
I run my own personal SMTP and IMAP server and haven't found it too hard to maintain after the initial setup phase. The main problem I had with getting emails accepted was my lack of reverse DNS PTR records on my domain. If you're unable to fix that (eg, you're not using a commericial internet connection that allows this) the solution is just to use another SMTP relay service. Some, like SMTP2go, are free if you're only sending a personal-use number of emails a day/month. That way you're still in charge of everything except the relaying of outgoing mail, which is easy enough to swap out.
I suppose I fall into the artisanal category (postfix,dovecot,spamassassin,roundcube etc). In our case everyone runs mobile/desktop IMAP clients, web mail at least seems to be a "when all else fails" backup plan for quick emails.
I do have a specific beef with all the consolidated email providers. If one of them determines your SMTP server to be spam (false-positive), ALL of their clients now reject your email with little recourse for the admin. Just had this happen with a solution that rhymes with 365. Even their clients were clueless as to how to resolve it.
i kinda have to disagree. i decided to take the plunge and i used a cheap racknerd vps and used Mailinabox. 5 minutes setup and occasional updates.
i got set as spam by google for the first few months but nothing since then. sure, one day i managed to spam like 200 emails in quick succession and that put me into spam but a quick "please select as unspam" solved it.
i've been running this for like a year and it has been a good experience. i recommend people try this out, it doesn't cost a tonne
Not the best choice of title. I didn't realize until I got to the end that this was about large organizations. No doubt for a large university or company this is true.
I would agree that setting up a robust personal email system is difficult and may be exceedingly so for an organization of any significant size but I am not sure I would attribute that to the quality of services from large providers. Our organization (a university) outsourced its email to MS (outlook) and it is simply awful. No IMAP or POP, no forwarding (this is of course the policy of the university, not MS' fault per se), important emails getting lost as 'spam' (including some once in a lifetime conference invitations) just because they came from the outside. The search is a complete nightmare. No way to send mass emails to one's class with, say individually generated temporary passwords, which I can easily do from my own server. The interface is clunky to say the least. In comparison, even Thunderbird shines (although why TB cannot implement its quick search function for years now is beyond me). So it is not the quality that the big providers supply. Security, maybe (even though they are a large target by default) but not convenience.
I suspect OP’s organization disabled it. Not really Microsoft’s fault but that how a lot of Enterprise products get a bad name - user hostile configuration.
As I mention in my post, this is the setting our admins chose so I do not blame MS for this (they support both, I know). Another setting they chose is to delete emails over six months of age and there is nothing I can do about it either.
The search and the interface are entirely the fault of MS as are the lack of more subtle features such as mass emails (which I have to use often while teaching online).
As is sharing links to OneDrive content with a password on it: https://support.microsoft.com/en-us/office/share-onedrive-fi... ("Set password: lets you set a password to access the file. When a user clicks the link, they will be prompted to enter a password before they can access the file. You'll need to provide this password separately to anyone you want to share the file with.")
(Poor quality sluggish Outlook client, dropping important emails into Spam, not being as configurable as a custom mailserver, hit-or-miss search results, those are all things I can agree with, I'm not just defending it).
Thank you for the links but just to clarify: I have to send each student a random individual password (so that they can login to set up their test taking software). The password is currently generated by a script that then proceeds to email it to each student. Sometimes I have to send individualized assignments (also randomly generated by computer algebra software). My personal email server makes it straightforward but I do not see how it is possible with MS. I guess a cleverly written IMAP client can do that for me but I cannot use IMAP, alas. The university policy is basically 'disable everything that can make using outlook bearable' (this last sentence should compensate for the ambiguous grammar you pointed out :)
They are going to block standard authentication soon though, leaving only their 'modern' webbased authentication. So basically IMAP/POP as we (and our mail clients) know it will no longer work with O365.
They've already delayed it a few times but they keep pushing for this.
I just did a ctrl-f here for "IP space reputation"
not mentioned yet?
One of the very important things is choosing what ISP to host your self-hosted email at. And the spam blacklist (or opaque/impossible-to-know) likely blacklist status of your IP at things like office365, gmail, etc.
Assuming for a moment that you are a person who is perfectly capable of setting up your own postfix and dovecot server.
No matter how perfect your rDNS, SPF, DKIM, DMARC setup is, and how flawless your theoretical postfix or other smtp daemon configuration is... If it's not hosted in the right place, outbound mail deliverability is the main problem you'll run into.
For the persons who are not ready to host their own SMTP and mail storage, I'm going to second the other suggestions made in this same thread that say a good first step is to control the authoritative DNS for your own domain, so that you can choose where to point the MX records at, and make an educated/informed choice of third party mail service provider.
I just did a ctrl+f for Yunohost and only got one hit! Have more people tried it? What was your experience like? Hosting my own email will be a 2022 project of mine. My current plan is to get like a linode/personal server (my first test for myself is to host my own html website :) yes I'm new) to avoid customizing my home router too much.
Not to accuse any one person specifically of anything nefarious, but I do find the historical timing of these articles bemoaning self-hosting of email rather odd.
Currently, there are only a handful of large technology companies in control of most of the world's inbox. Google is the first that comes to mind. At the same time we've just had some of our most highly publicized hearings involving these tech companies (facebook and google) as well as our first hearing on cryptocurrency and the larger web3 infrastructure. There is a rising public awareness and therefore political will to regulate these technologies and companies. This would formally fold google and the like into the USG despite their long standing less formal arrangements with the intelligence agencies.
The one way to make this regulation and upcoming legislation moot? Decentralization by any other name. Self-hosting of email servers, or distributed computing and storage with web3. Perhaps its is only my latent paranoia, but I can't help but shake the feeling that the glut of the 'don't bother hosting your own email server' sentiment is, at least in part, artificially amplified in order for the coming formal regulations to have more of an impact.
It wouldn't surprise me at all if google themselves was helping to facilitate this in order to steer conversation towards stalling any potential competitors as a part of the new regulatory framework. It is for instance a lot easier to argue that outlawing self-hosting of email servers (or requiring a license to do so) makes a lot more sense if you can point to a 'general public sentiment' that hosting your own email server is 'too complicated' and 'less useful' and 'less secure' and therefore would only be done by antisocial actors such as 'criminal elements' and 'terrorists'.
Seeing as we are at the cusp of a new distributed infrastructure movement, all this feels like preemptive damage control to me.
People trusted that e-mail offering by Google would be compatible with an existing e-mail ecosystem.
Google deals in information control with e-mail hosting being one of tools. They are also very efficient at presenting themselves as being right. So folks and companies having trouble sending messages to GMail, moved to GMail over time since it seemed doing good job, in opposition to their sending offering which seemed at fault.
Google seems to be incapable of not fucking up anything that disrupts their ability to spy. I believe that as soon as they figure out a better way to creep on people's finances and purchases on a global scale, they'll move on to killing e-mail globally.
Microsoft, in contrast, seems to be incompetent when it comes to e-mail. From my perspective this was a true statement since they touched e-mail. 'member Outlook Express? Dealt with Office365? Same shit 25 years apart.
I also have my own domain name and whenever I sign up for anything (if I really have to give them an email address) I will make it: companyxy.net@mydomainname.me and indeed, whenever I receive a spam email I know what company sold (or leaked) my email address, because it is right there.....
I was doing well with my own email server until I got on some blacklist from Microsoft, all my outlook.com contacts didn't receive my email anymore, not even in their spam folder. I pushed hard on support, eventually they told me they don't control the process ??
I gave up. In retrospect I should have used another smtp gateway (perhaps from my provider), but back then the thought never occurred to me.
For a time it was nice, (basically) unlimited storage, unlimited aliasses, being able to send 200 mb (only to my wife admittedly) easily. I did really learn a lot, which it what I tell people now: Try and do it to learn, but it's really unpractical. I now pay for email.
Last time I hosted was ~2004. I had a very tight qmail implementation, but even then I seem to be getting black-holed or marked as spam. Maybe it was because I was self hosting on my local home ISP and using dyndns to get around potential changes to my ip address from my ISP. Maybe spam filters didn't like inconsistent IP addresses? I don't know. But asking people to whitelist me was a hassle.
At the time I was a heavy Mac user though so I ended up switching to mac.com email, and then Gmail when Apple EOL'ed Mac.com.
Today though there's probably an easy turnkey VM or docker image well-configured to work without too many false positive spam flags... I hope?
I think you can extend this a whole host of other domains. A lot of the bread and butter of 2000s era IT departments are being replaced with a highly competitive SaaS market. Building brochure sites, email, CRM, CMS, e-commerce, directory services, accounting and ERP. Twenty years ago you'd do a months-long procurement process for enterprise licenses, set SLAs, size the hardware, buy servers, hire integrators and you'd get your software up an running in 8 months if you're lucky. Now you type your credit card into a website and get unlimited capacity and 99.9% uptime instantly.
The latest edition of Nemeth, et al., Unix/Linux System Administration says basically the same thing. Spam filtering, for example, is a whole horrible thing that will be essentially a full-time job for anyone running a mail host.
Hell, about two years ago I gave up on self-hosting. Unwanted email got to be the vast majority of what I was receiving and spam filter software ate more memory than my itty-bitty host had, meaning I would have had to get a bigger, specific mail host. (As it was, the IMAP server was the biggest process running on my host.)
As a person running his own email system, I think it's taking more efforts from people to convince everyone to don't run their email systems than managing it effectively, and I don't have any bouncing even towards gmail et similia. As a company the only concern for me is not managing an email, is more to give third party private for profit companies access to all my communications (even if I guess sending it to other unaware people using gmail / 365 has the same effect), but for me it's crazy that the world has accepted that
I run my own mailserver and I've been doing that for the last ~10 years.
I have a static ipv4 at home, and my biggest problems are the following:
- residential providers won't delegate a reverse dns zone or set a reverse ptr record for you -- or at least my ISP (Fastweb) will not do it for a residential contract
- spam lists (spamhaus etc) will blacklist residential ipv4 pools by default applying what effectively is a prejudice (and defamation).
Some considerations:
Major providers will effectively do as much as they can to prevent other organizations (let alone individuals) deliver their own mail.
Google is particularly shitty in this regard: it regularly delivers my mail to spam despite having both SPF, DKIM and DMARC in order. My gmail inbox however is full of SPAM because google decided that I really have to look at those promotional email.
Microsoft is surprisingly good instead: upon rejecting mail initally they're going to direct you to an automated procedure to de-list your domain from their spam services, and it works. Kudos to microsoft.
SPAM, surprisingly, isn't really a problem: if you have sufficient checks for incoming mail (does SPF for the domain allows this ip to send email? do DKIM signatures check out? are they using SSL/TLS for their connection? et similiar) you basically won't receive spam.
Most OS vendors (Red Hat / Debian / Ubuntu) deliver postfix with poor cryptographical default settings, meaning that the default settings will connect to plaintext SMTP to deliver outgoing email and will not setup submission (tls) by default, not even with a self-signed certificate. In the time of letsencrypt being available, this is a dumb choice.
Running low-volume mailserver is surprisingly light on resources. A raspberry pi is likely overpowered for the task. You can use pretty much anything and it's going to work, as long as it powers on and doesn't lose data.
Running a mailserver is also surprisingly versatile. The possibilities are pretty much endless. You want to alter outgoing emails? No problem, look at PCRE maps and postfix's header_checks (or similar). Want to have mail aliases? no problem. Want a catch-all address? easy. Want to hook a service into mail delivery chain? look at the milter protocols.
FTP/SFTP are still very much in use today. Removing support from the browser actually makes sense to me as the majority of browser users are not using it for FTP. There are a myriad of FTP/SFTP clients that are vastly more powerful and have more security options than any browser implementation of the FTP client ever had.
As far as spam filters go, too many good emails end up in spam folder nowadays. It has become a common place - "check you spam folder". I find it's easier to have a few manual filters and just delete all the remaining spam manually as it arrives, because the alternative is to check two folders every time - Inbox and Spam folder, with all the spam still there of course.
Running your own email is impractical because it's the only way for many people and services to get in contact with you. You really don't want it to fail. And it's entirely on you to ensure that you have monitoring to notify you when it does fail (through some non-email mechanism). And you better be prepared to solve the issue for yourself in a timely manner.
The best way I have found to run an email server is to use it to receive emails only and to get your own emails out send them directly through a provider like Amazon SES. Now you can create all of the email accounts with as much space as you want but still get your emails delivered. It is not ideal but the only practical way to run a small email system.
But I think you can set up an email server for your own tasks, like sending yourself some notifications. It's also nice to know how this works in theory.
These are the reasons I included some basic e-mail setup in my book (Deployment from Scratch), although also advising to depend on some reputable IP addresses.
I don't know. Cpanel makes it almost effortless. There are rare problems, and they don't tend to be difficult to fix. If we were using a provider such as Mailchimp, for our marketing email, we'd probably be spending $30k to $50k a year on email. It's well worth the very minor effort required to host our own email.
email is totally broken. this article is right but the answer isn't let goog read your messages, it's invest in new communication protocols
tons of serious players don't trust email at all today
some, like my health insurance + bank, are dinosaurs who are all but licking the molten shockwave of a meteor. But still -- they have real reasons they email me that my 'secure message inbox' has a new message. Oscar uses 'secure email powered by virtru' to tell me they're not going to reply to my reply to their email, wtf, huge indictment of oscar, but also indictment of email.
but it's not just dinosaurs -- amazon, who has had a tough month but is generally savvy, doesn't trust email. they won't send me itemized receipts because they don't want google to read them.
I half believe we're about to see a renaissance in self hosting for individuals + businesses. this article is calling the bottom.
I'm a well versed software engineer and even I had trouble setting up my email. After a while I realized it was not worthed, so I keep using Gmail, hoping I don't violate any of their policies or get tangled in the algorithm. As I see here, it's a hard thing to recover once this go wrong with big tech.
Frankly no. Both Google and Microsoft office have taken a massive step back by implementing "smart" functionality into their mail search which makes accessing critical information (and therefore my job) impossible...
I'll be self hosting until these companies return basic working functionality
Fastmail is, IME, an excellent email service provider. No, it's not really "running your own", but it's exactly what most people who want to avoid gmail are looking for, re: deliverability, featureset, privacy, UX, etc. Highest recommendation, from an unaffiliated happy customer.
Is the article sponsored by Google or Microsoft? We're not in 1995 still, and I would say that it has become practically more doable for each year, given the affordable (sometimes even free) and easy access to software, hardware, and internet connectivity required to put up an e-mail server.
>This is not directly about the big providers making it harder and harder to send them email, although that doesn't help. It's because a quality modern email environment is big, complex, and takes a lot of work to create and keep running.
The main problem I've run into while running my own email server is IP reputation issues. It is still an ongoing issue for me.
You can read my previous IP reputation issue in this comment [11].
You should make sure your server's IP address isn't blacklisted. If it is, you're going to have major delivery issues with some email server providers (ESP). Some blacklists you can check are listed here [0]-[3].
I think my main problem now is the UCE Protect [5] blacklist. I think some of the major ESPs use their harshest blacklist from UCE Protect, which is their Level 3 blacklist [6]. This blacklist will include your IP if your ISP meets a spam threshold for any of their other IP addresses. This makes running a mail server on cheap hosting providers like Digital Ocean or Linode very difficult.
My conclusion is I should switch to a more expensive ISP that isn't in danger of getting on the UCEPROTECTL3 list or find an email forwarding service for a next hop destination for outgoing mail.
You can read more about UCE Protect here [7]-[10].
I use a mail.com email. I can’t count the number of times I carefully spelled my email on the phone “M-A-I-L”.com only for them to still try to email me (unsuccessfully) at gmail.com.
I mean I get it but it’s not good for the internet for people to equate all email with gmail.
I am very happy with my artisanal choice. I have been doing so for several decades, now. Usually only trouble when I have to move to a new IP range (changing colo provider), as it means re-establishing reputation and so forth.
I think I am showing my age when I say, I remember when Linux by default came with sendmail enabled by default. You could use your Linux box to send e-mail anywhere without it getting caught up in spam filters. Fun times.
I think you misunderstood. If you fire up a mail server on your Linux box today and try using it for mail the big mail servers are sending your mail to spam due to a bunch of spam rules. These days, you need an organization with a known reputation for sending good e-mail to handle your e-mail so you don't end up in a spam folder. It's how the big players stay big.
One of the big email providers that starts with a g has a really bad outgoing spam problem through their cloud api. Hard to block because you can't block them or you will miss real emails from them.
> Your IMAP access might be as good as theirs, but things like your webmail, your spam filtering, and almost certainly your general security will not be as good as they have.[...]
> Especially, open source can't compete on features like webmail and performance in things like spam filtering.
If anything roundcube is a better webmail client than many of the mail-provider ones. And that's ignoring all the ads and tracking that these come embedded with, even if you pay for them.
I haven't used gmail-for-organizations but if it's anything like the normal gmail interface then ... I guess some power users will prefer it? But in my experience many people prefer roundcube, because it's simple and usable. Not that it's perfect or better in all the ways, it's just from my experience and the users I talk to, it is just as good and fills a need that gmail doesn't.
Same goes for spam-filtering. It's not that spamassassin/amavisd/rspamd/postscreen/RBLs/whatever is 100% perfect, it just get's you pretty far, and from my experience also gmail, as the main contender, has varying success on how close they achieve 100%.
And even security is not magic. A large mail provider doesn't have access to magically different security tooling than everyone else. They have a threat model that is slightly different and their scale allows them to do some things that not everyone can. But wrt to one's userbase it's perfectly possible to be "just as secure".
Running your own org-mailinfrastructure is certainly not "artisanal" - for some reason this comes off slighly dismissive in the article - it's just that, as anything, it's work that you have to want to invest in. A trade-off where it often does make sense to outsource. But then email is not so different from any other service you want to provide.
... but then I see that the article seems to partially be writing off of the experience of using the U of Toronto mail system, which seems to be using squirrelmail and procmail. I didn't even know squirrelmail was still developed - this impression kind supported by there being no news between 2013 and October 21, 2021 on the frontpage https://squirrelmail.org/ while procmail is unsupported since approximately forever (it feels like pre-9/11 but I am not sure).
If you compare that experience with outlook.com - then I can certainly see why one could come to the conclusion in this article.
EDIT: Heh: my guess of procmail being pre-9/11. Wikipedia says: "Final release 3.22 / September 10, 2001"
This is completely wrong, SMTP is decentralized. Own your future, self host your mail, it's not hard. I predict that eventually all messaging apps will use SMTP under the hood! Xo
simple-nixos-mailserver is a really nice way to get up and running. Postfix/dovecot/rspamd/autocerts+renew/clamav/dkim/managesieve
Super easy to set up and nothing in the article will be a problem. Your system will not be "artisanal". 400 people wasting time commenting on this guys dumb article.
Personally, I use email less and less, thanks to Matrix. I wish websites would stop requiring an email address for signing up though (and phone numbers for that matter).
Feels like being able to send mail from an IPv6 address would mean no one shares a sender IP again, and everyone's sender reputation gets a fair shake.
Minimum requirements:
Static IP(s)
DNS - A and PTR
DNS - MX
DNS - SPF
Nice to have:
DKIM
DMARC
Get all that in line and it should work fine here. I run quite a lot of email systems. I run them in the UK and not Canada or anywhere else so can't comment broadly.
I've been doing this for 25 years now, so I think I have a fair handle on how email works.
You can stuff your artisans up your arse. Email is email - do it right or fuck off.
[EDIT: I give up, I can't remember how to format a list on HN and I can't be arsed anymore - I created two lists above - they should be fairly obvious and each one has a heading followed by a colon]
It's only probably your domains already have enough reputations.
I don't think you understand the complexity of having your emails marked as spams on newer domains.
I've also been running my domain email for 15 years or so and configurations are easy but I still rely on external delivery services just so I get less chance of being flagged as spam.
This has been true for a very long time. Spam really destroyed SMTP as an open protocol just like it did for all the other old-school federated protocols.
I am old enough to remember when the internet was about open standards and everybody could run any service including email servers without much hassle.
It's obvious when your domain (well, the IP really) is blacklisted. You either get a nasty error from the mail system saying it didn't go through or that the server was unreachable or what have you. You're always going to get some notice that it failed. In that case I contact the recipient a different way and say, "hey, you're mail server is broken, can you please fix it?" Usually this means them contacting their mail provider and asking why they are blocking mail from someone they need to communicate with. If it's someone like Google, they'll realize they can't actually get any support and think twice about using Google for mail.
A lot of people will say, "no they'll blame you for it not going through", but that's rare. Most people will be receptive to your insistence that you're trying to send them mail but their provider is in the wrong.
Now, if you mean blacklisting in the sense that the server has shadowbanned you and is sending back "221 OK", then, again, you have an affirmative defense: "hey, you're mail server said it accepted the mail, can you please check with your provider on what they did with it?"
In either case, this is actually not a good thing for the recipient's mail provider especially if they pay for that provider. "Why did you accept the email from the sender but not put it in my mailbox? Who else have you done that for? Why am I using you as my mail provider again?"
I have a circle of influence... about 100 people. All 100 of those people would switch mail providers if I asked them to. And I think a lot of people have a circle of influence around this size as well. So there actually is an amount of control over these bigger mail providers. They will be receptive to "I just told your customer to switch because you won't let me email them" especially if it's widespread. In exchange for that "power", I make sure my mail server is as clean as possible and quickly respond to any notices sent to my abuse@ address.
I advise everyone to take a look at the EU Digital Markets Act. This may be a unique opportunity to force large corporations into behaving nicely with others on the interoperability front and kill off their malicious monopolies at the root.
> It's cute, decent enough, and hand-crafted, but it doesn't measure up in usability, features, and performance to the email infrastructure that is run by big providers
What a BS, wake up dude.
> usability, features, and performance?
you really make me laugh, it's so bloated it's barely usable.
It is a wonderful world we live in. Everything from salary negotiations to love affairs is on someone else's servers, all in plaintext. Apart from email this also applies to Slack, which is a goldmine for keeping dossiers on developers.
This is one thing the authoritarians like Biden (Clipper Chip, Patriot Act) won't want to fix. There will be no law that companies with more than 100 employees must accept mail from individual servers (they would still have the correspondence anyway, but it would be a start). There will be no law that all mail must be encrypted.
I strongly agree on the point that running your own email service in 2021 is a pain in the a*s.
I strongly disagree on the motivations given by the article (the service you provide will never be on par with Google/Microsoft). This is plain false. There's plenty of webmail clients that look as modern as a Google/Microsoft UI. I even find the Outlook UI to be more primitive and spartan than many open-source UIs.
If running a mail server in 2021 is painful, that's because cloud giants have built a small mafia that, using the excuse of spam protection, excludes anything that doesn't come from a handful of mail services, period.
I used to have my own @mydomain.com mail server back in the golden age of the Internet, with mydomain.com pointing to my sitting in my closet. At some point, around a decade ago, mails sent from there started being refused - Spamhaus really doesn't like SMTP servers with dynamic IPs.
Fair enough. Back then I migrated my home server to a Linode instance. It worked for a while, but then even those emails started being sent back by some providers.
Fair enough - I thought. In theory anybody could still rent a Linode instance, buy a domain name, set up their mail server, and spam the world.
So I moved @mydomain.com under my managed (paid) ProtonMail account, installed a ProtonMail bridge on my server, and forwarded emails this way.
This should be alright, isn't it? I'm paying for a Linode instance connected to my name and surname. I'm paying for a domain name connected to my name and surname. I'm paying for a ProtonMail account linked to my name and surname. What prevents me from sending emails from my own server, using my own domain, my own logic and my own rules? There are so many ways to get back to me as a person in case of abuse, so the risk of spam should be low, right?
Unfortunately, filthy Microsoft thinks differently. Any email sent to @live.com, @outlook.com, @hotmail.com etc. gets sent back, blocked by Microsoft's spam filters. Even if it's sent through a legitimate paid domain name, from a legitimately purchased cloud service, through an account connected to a legitimate and paid ProtonMail account.
So eventually I've resorted to Mailgun to deliver my notifications.
Do you see the sad irony of this situation? Owning a domain, a cloud server and linking your account to a legitimate mail provider is no longer sufficient for you to deliver emails. If you don't want to use a Google/Microsoft/Apple account, then you have to pay extra bucks for a service like Mailgun, whose sole purpose is to do what SMTP servers have been able to do (for free) for the past 5 decades.
So, in a nutshell, running a mail service in 2021 is not difficult because it's hard to set up or it can't compete with the major players. It's difficult because a small mafia of cloud providers have done their best to make it difficult and force you either to use their services, or spend money for doing something that people have been doing for free for decades.
Email was supposed to be an open protocol, and barriers to set up your server used to be low. That's no longer the case. With the excuse of greater security and less spam, a small bunch of companies has built a criminal organization that has turned email into a closed standard, with Google and Microsoft alone running more than half of the traffic in US and Europe.
Shame on us all for giving up such an inalienable right of the Internet without even attempting a fight!
Do you have ~30,000 EUR / year for skilled admin? Plus ~20,000 EUR / year for hardware and other running costs? If you do, you can have your emails safely and reliably exchanged from your basement. If you don't, you can rent whatever on the 'net that suits your budget.
This is not true. Regardless of how secure and properly setup your server is:
Outlook blocks anybody that does not send enough mail, even if you've never sent any spam and are on a clean block. They're happy to let their users to send you spam, but ironically they still block you when you you try to report it to abuse. The good thing is they block you, so the email bounced and you know it wasn't delivered.
Gmail classify messages to a user who has never communicated with you before as spam. This is silent, so you never really know if an email to a Gmail box has been filtered out as spam or not. Their abuse inbox accepts messages but I'm not sure they do anything with it.
Basically, email has been hijacked by two companies.
I think those numbers are too high. Even if not, companies like Airbus have that money and are using G-Suite. So Google has all corporate information (does the swamp redirect some of it to Boeing?).
In my experience the time sink is high when you're setting it up. Then it's mostly zero.
Though I'm not sure I'd recommend to invest into it. Back in the days that unix knowledge was valuable and setting up your own e-mail was a good way to learn thing or two. These days those skills are useless for most people, so I'd say use hosted mail and spend that time learning some more valuable skills.
Wait, is email spam still a thing? I only get a few spam a day and Spamassassin easily identifies them.
I had assumed that the spammers had moved off to other mediums. Either that or they are specifically targeting big servers like Gmail and are leaving the smaller servers, with their varied (artisanal) anti-spam approaches alone.
It never stopped. What is however more annoying are all the "marketing messages" from all the companies you ever bought a single thing from in your life. It's incredible how often some companies spam you with this (often more than once a day). Really not sure what they are trying to achieve, but for me it's the resolution never to buy anything from them again as I filter their whole domain permanently.
Those I usually forward to their legal department with a little extra bit that says that I have stopped doing business with them on account of their UCE.
A lot of it is luck. You can go years not seeing much of the really malicious spam and then one day you're on the list and you'll find yourself being bombarded with stuff Spamassassin doesn't touch.
Especially if your organization is a potential financial target.
If anything, it's worse than ever. The favored technique these days is "snowshoe" spamming, where $SPAMMER sends a trickle of spam from a large number of IP addresses. About the only way to stop it is to block the /24, and then they just move on to the next block of IPs they want to ruin. RBLs like Spamhaus are helpful, but there will always be a few spammer IPs that haven't yet been listed. SpamAssassin can be handy, but it's a pain and there's a lot of rope you can hang yourself with. Blocking certain TLDs outright (.cam is a good candidate not just because of this, but also the phishing potential) can be an option.
Even a service as big as Microsoft's has pretty bad spamfiltering though. It's slightly better than an untuned SpamAssassin config but it's really not a lot better.
Another reason could be that I live in a country with very strict anti-spam and privacy laws (Canada). I have always assumed that spammers wouldn't care but who knows...
- Getting your own domain and using a provider such as fastmail or proton is a first step that gives you lots of security fom arbitrary. Because you own the doorstep, you can change provider without having to inform all your contacts of the move. You're also more secure from unilateral moves from your provider.
-Hosting your own mail server means that you are responsible for the persistence of your mail. It's a nice artisanal thing to do, and you may be satisfied to know that no one is reading your mail.
-Sending your mail yourself is the real hard part, because you need a stable IP that is accepted as a legitimate mail sender. Moreover, you need to monitor this property in the long term. Every mail server has their own way to choose who is a legitimate mail sender, and it's an ongoing pain to check that.
You're not forced to go all the way, you can simply pick a domain to secure the frontdoor, or you can host your mail server without sending mail by yourself, etc. You can also self-host, and change your mind later without much impact.
I personally would incite everyone to do at least #1 for safety reasons, #2 if you want to fiddle with the system to know how it works, and to avoid #3.