What we need next is a widely adopted standard of end-to-end opportunistic encryption, similar to Signal. PKI and PGP are too complex for end users.
The problem that prevents such a standard from ever taking off is that, unlike TLS, there is zero interest from major providers to push end-to-end encyption, because they absolutely need access to the plain text for commercial reasons. So they advertise STARTTLS as "email encryption". By that token, CDMA and GSM are "email encryption" as well, it's not like they push the plain text naked over radio.
It's not a standard, but it's there: autocrypt (https://autocrypt.org) aims to make end to end encryption as easy as possible. It's using PGP underneath but the whole idea is to hide key management (the real pain point) from the user. It's a long term initiative, divided in multiple levels of "compliance" that gradually break compatibility with existing systems more and more. Level 1 is there already, with some implementations.
They are also the only possibilities to provide proper end-to-end encryption. The last thing we need is another proprietary “standard”
(remember how those same companies pushed for drm in the w3c spec).
Yet the only real change can come when end users understand the implications of using the free email providers. I think this is harder for most than using a pgp email client plugin.
Sure, STARTTLS is better than nothing, but your email provider still reads your email. When we talk about email encryption, we mean end-to-end encryption.
> STARTTLS is not an end-to-end encryption system. [...] But for various reasons it’s hard to deploy end-to-end email encryption, and we’ve spent decades trying. Also, STARTTLS works just fine with end-to-end encryption.
> Please indulge me: I think a small rant is appropriate here. There are some security specialists who think that only the perfect is acceptable. Nonsense! Requiring perfection is crazy. [...] For almost all users, email encryption with STARTTLS is a major improvement over what they had before. Let’s keep working to deploy even better systems, but let’s take partial victories where we can get them.
I am my own provider. It's really not that hard to configure a mail server. Other peoples' providers, however... so I tried using PGP. It's somehow manages to be more difficult to use than configuring and running your own mailsystem which doesn't deliver straight to gmails' spam folder, but I digress; but then my emails are at mercy of the end devices: who knows if outlook doesn't send a plaintext email home as "telemetry"? Same thing about chrome with all the extensions for using PGP... I mean, security is hard as-is, but security of an ubiquitous thing, especially when people have been conditioned to undervalue security and privacy, is a lost cause. It's best to treat emails as being in the same category as phone calls, conversations in subway and sms'es - public.
> I am my own provider. It's really not that hard to configure a mail server.
Oh yes it is. I mean for professional sysadmins it may not be, but try talking any normal person through buying a domain, getting an SSL certificate, attaching that at your MTA, configuring all those fiddly DNS records that prove you're emails are who you say you are. And all of that isn't even taking into account how needlessly painful the actual process of just installing the MTA is.
And what do you have for show from all of that? Still no real assurance that half your mail won't end up in spam, no web front end (that step alone adds several additional layers of complexity), less guarantees regarding up time (since you no have a single server that you need to manage yourself), arguably worse security (since you're now asking a layman to harden a Linux box) and the whole thing comes with a heftier price tag than Gsuite.
We're not talking about self hosting an IRC server or other service you can throw up in an hour and then forget about. Email is a frustrating sequence of a multitude of steps where even just one of them could render your set up worthless. Hence why even many of us sysadmins have long since given up self hosting email servers. It's just not worth the effort
I consider myself very technical, and know my way around Windows and Linux pretty well (I've used both for many years), but setting up a mail server is something I do very infrequently and find complicated to get right when I do so, especially when getting it working with TLS, DKIM, SPF and spam filtering.
No, in my experience, selectodude is right. I've written this rant before but I think it's worth sharing repeatedly, so here goes again:
The big players (Google, in particular...in case it's not clear, the bulk of this rant is leveled at Google; Microsoft is an offender on perhaps 25% of the level, Yahoo just doesn't seem to care) have ZERO interest in "playing nice" with small e-mail servers. My experience running my own e-mail for almost twenty years then finally throwing in the towel and migrating to Fastmail not only proves this for me, but is, according to other sysadmins I've talked to, quite common.
I did everything right when hosting my e-mail. I did it for myself and maybe four others. The volume of mail going out of my server was so low, I could read the outbound mail.log for myself, and routinely did. Every domain had SPF, DKIM, and DMARC records. The outgoing single IPv4 and IPv6 addresses I used had perfectly-matching forward and reverse DNS lookups. I had a script that checked every blacklist I could find every 24 hours. My IP addresses were allocated from my (professional) colocation provider and SWIP'd to me with valid contact information. My IP addresses had been unchanged for seven years and had been stable for ten years prior to that change. My colo provider had a draconian non-spam policy. Nobody ran an e-mail list or anything. I've used the exact same domain for e-mail for over twenty years.
Still, every couple of months, Google (sometimes Microsoft, but far and away most likely Google) would decide to simply stop delivering messages from my five users--remember, including me--to Google's users. The messages would just...disappear. Sometimes, Google would stop delivering the messages mid-conversation. Replies would be going back and forth but then my reply to a Gmail user would go to /dev/null while their messages to me would still arrive. Most of the time, the mail would go to spam (even with that little yellow tag meaning "verified sender" or some such) but 20% of the time the mails would just disappear.
Worse, the destination would say "250 OK" to every e-mail I sent them so I had no way of knowing that the messages had been sent to the big bitbucket in the sky. I got annoyed.
I signed up for Google's Postmaster Tools and Microsoft's similar offering. Microsoft would at least show me telemetry but it wouldn't do any good. Google? Google would proclaim that I wasn't sending them enough e-mail so I couldn't show up on the dashboard.
Finally, after three years of fighting this, it came to a head when I was, at all places, at a boardgame meetup during one of these "blackout periods" and got introduced to a Google SRE who worked "adjacent" to Gmail. "Oh, hey, you should talk to techsupporter, he does e-mail too!" was the opening line. We talked, I ranted, and he actually said he'd try to help. He escalated internally and the word came down from on high on their SMTP team or some such: "domain [mydomain] has insufficient reputation."
What did that mean? Apparently, their system--at least at that moment--thought my domain was "too new" and they tilted towards just binning the e-mail. I got red, saw stars, then gave Fastmail $390 for three years of service on three e-mail accounts and moved everything to them within five hours. That was the last straw, the final injustice. For me, the "decentralized Internet" died on that day.
Why was I so pissed? Observe these dates from WHOIS:
I wish I could upvote you more than once. No matter how perfectly your mx environment is set up, the IP space internal reputation tools run by Google and MS for office365 will remain totally opaque. I have this issue occasionally with o365 hosted destination smtp, and I control my own ASN and ARIN IP space. Not once in the history of mankind has an open relay run on my IP space, nor has spam come from it.
> The volume of mail going out of my server was so low, I could read the outbound mail.log for myself, and routinely did.
This was your actual problem. The large providers (gmail, outlook, yahoo) and appliance vendors (ironport, etc.) are all now treating "unknown" senders as spammers. If you don't send a large enough volume of emails, you will never make it into their reputation tracking lists because "it's not worth it". End result, you are considered unknown and will be treated as a spammer forever.
It is a huge organically grown cartel where the protocols work but you cannot enter a small player, no matter how well you behave. It is a shame, but there's nothing you can do about it :(
"Worse, the destination would say '250 OK' to every e-mail I sent them so I had no way of knowing that the messages had been sent to the big bitbucket in the sky. I got annoyed"
This. I go to a lot of trouble to follow all relevant RFCs, SPF, DKIM, DMARC, and TLS. My intent is to help other servers know I'm transmitting in good faith, I'm hopes I'll get the same fair shake.
If it's still not good enough for you, please at least let me know that fact, with a message I can investigate. At the very least, don't outright lie, which is what I take a bogus 250 to be doing.
I don't think you read or understood what I wrote. You were not routing through a smart host (third party SMTP relay.) This is your problem.
There is a specific reason I recommend doing so: the reputation of the sending IP address almost always overrides the reputation of the sending domain.
A smart host operated by a third party, such as Google: https://support.google.com/a/answer/2956491?hl=en , would have solved your blackholing issues because it's the third party's job to maintain extremely good reputation of their IPs sending mail. Most importantly, they have high volume of legit emails, and volume is key to maintain good reputation. Finally, even if you have a low-volume domain, your emails will benefit of the good rep of the smart host, and thus would not be blackholed.
I will repeat it again: if you host your mail server, you definitely should also use a smart host, and all the spam/blackholing issues disappear.
Using Gmail as a smart host defeats the point as mentioned in the root comment of this thread: Google can read your email.
Google also rewrites the "From:" address to your Google account's address, unless you jump through some hoops to add it to your account. This has become more difficult recently, and needs to be done for every address (which is an issue if you use single-purpose email addresses).
I agree this doesn't sound nice, but for years before "cloud", most orgs had exchange servers and we rarely had this problem. I think it's still possible to self-host.
Not if you send your mail through a smart host (an outgoing SMTP proxy) which is in good standing. Most sane ISPs/IAPs require this anyway. I've hosted my own mail server since about 1996, moving from Sendmail on a 66MHz Pentium hanging off a cable internet connection to the current Exim on an Intel SS4200 on gigabit fiber - none of it high-powered hardware as such is not needed for the purpose. Mail sent from here arrives just fine, no problems there.
Not true. This might be happening if you are trying to host a server on your domestic internet connection, those IP ranges are usually banned in spam filters. But you can easily host your e-mail server on a real datacenter server or something similar, and you'll have almost no problems with spam filtering. (In fact many companies and even individuals already do this and have done this for long time.)
Honest question: Where do people get this idea that they state so confidently, that it's "impossible" to run one's own mail server? Have they actually tried or are they just repeating what they've heard?
Counter-anecdote: I have run my own mail server for several years. I use it for all my email communication, and also send out daily emails to a couple of thousand subscribers[1] who are on all sorts of email providers. The "canary accounts" I have set up on all the big providers all seem to have their subscription messages delivered straight to the inbox just fine, though I should probably monitor this better somehow.
> Not true. This might be happening if you are trying to host a server on your domestic internet connection, those IP ranges are usually banned in spam filters.
This is pretty much what "hosting your own" really means. Having SSH access to some virtual machine somewhere is much better than letting gmail read all your mail, but it's still isn't self hosting. You could still use the remote host as a mere relay, but it's still not ideal.
Real self hosting means having your box at home, sending and receiving mail directly. And that is effectively impossible. Self hosting is dead. I'm not sure how we could resurrect it.
Host at home, relay via an IP address owned by a hosting provider. Pragmatism meets idealism.
Of course, I don't do this because I don't want to rely on my consumer internet connection or domestic electricity provider, and my flat is too small for a rack anyway. Also the noise, electricity bills and waste heat play a part.
Some things are best done centrally. I am happy with servers in the cloud, I really don't miss having to travel half way across the country to replace a hard drive...
> Host at home, relay via an IP address owned by a hosting provider.
As I said, "You could still use the remote host as a mere relay, but it's still not ideal". And by the way, as much as I'd like to host my email at home, I still gave in somewhat and use a remote virtual machine (at https://gandi.net).
> my flat is too small for a rack anyway. Also the noise, electricity bills and waste heat play a part.
You need less than 5W, a small shoe box worth of space, and no fan at all to host your email at home. R-Pi, Sheeva plug… A web site with moderate traffic might be more problematic. But we're nowhere near a rack. For remote hosting, you don't need more than a fraction of a CPU core.
Reliability, though, that can be a real problem. And a big reason why I still use my virtual hosting provider. I'd love a redundant setup, where if something goes down, the backup takes over, and notifies you about needing to replace a piece of hardware. (Of course, it should be plug & play so my grandma could do it… one can dream.)
> (In fact many companies and even individuals already do this and have done this for long time.)
Yes, with mixed results.
For one instance, see the trouble Linux people are having with getting messages delivered to Linus' Gmail inbox. I recall reading that some have needed to sign up for Gmail specifically just to get their patches to Linus. Plus, the 2015 occurrence with Gmail having a 20% false positive rate: https://plus.google.com/+LinusTorvalds/posts/DiG9qANf5PA
> Not true. This might be happening if you are trying to host a server on your domestic internet connection, those IP ranges are usually banned in spam filters.
As are many cloud hosts / VPSs. AWS, OVH, Digital Ocean, etc I've heard plenty of horror stories from people trying to run mail servers on all kinds of cloud platforms (and personally experienced issues on EC2 instances myself)
So you are saying you do not trust your e-mail provider to secure your email. At that point I suggest finding another provider or running your own service. This secures a significant point of interception and modification of data.
In general, if a government asks your provider for access to your emails then will hand them over. The only way to mitigate this threat is to use e2e encryption or host the email server on premises but then you start encountering deliverability issues.
The Internet is full of stories of people having trouble with large email providers (notably Google and Microsoft) accepting self-hosted users' mail. A self-hosted setup's IP starts with 0 reputation (or worse, depending on who owned the IP address beforehand) and will face throttling and outright blocking for a painful amount of time.
Furthermore, if you forward your email to e.g. a Gmail inbox, any spam sent to your address will go towards lowering the reputation of your forwarder, meaning that someone just has to send you a lot of spam to get your forwarder blocked from Google servers.
SPF, DKIM, DMARC and TLS all cannot mitigate the above.
Point 1: spf, dkim, dmarc is enough to get yourself neutral with gmail. You have to do it from get-go tho, before you tarnish your reputation. Being in DNSBL lists does not help either. Set up proper reverse dns entries also. Tls is completely optional. Haven't had any trouble neither with gmail nor ms using both residential and datacenter ips in two EU countries. I don't consider myself a hardcore linux admin specialising in mail, so there's that. As far as the second point, about forwarding spam goes, you're running a mailserver without spamassasin? What is this, the 1990s?
Counterpoint 1: I wrote my rant as a separate reply but the summary is that after twenty years of running my own e-mail and doing everything "right," I still got regularly binned by Google and Microsoft and (rarely) Yahoo, so I gave it up and switched to Fastmail. My experience, according to chats with others in my field, is not unique. I'd actually go so far to say that yours, with reliable delivery to Gmail with a small server, is the outlier.
> you're running a mailserver without spamassasin? What is this, the 1990s?
More like, relying on SpamAssassin to reliably and correctly classify spam is the anachronism here.
Even if SpamAssassin could do a good job of filtering spam today (which it can't), the set of messages that SpamAssassin classifies as spam will inevitably differ from the set of messages that Gmail classifies as spam, so the only thing this achieves is another point where you have to worry about false positives and training the classifiers.
Furthermore, the problem is exacerbated by poor support of forwarding in modern MTAs. They will queue all received messages first, and when Gmail rejects certain messages during the IMAP session (spam with high confidence, or messages it considers invalid), they result in a bounce, and the thus ensuing backscatter further degrades your mail server's reputation.
The closest thing to correct behavior is Exim's cutthrough delivery feature, which has been implemented "relatively" recently, and even so it is not complete and I've run into issues with it with non-textbook setups.
I'm trying to figure out how big those problems actually are. I can't say anything about the stories people tell, but I have no trouble sending to gmail nor anybody else.
Using configurations that provide redundancy is quite trivial. Also, what is the point of having several nines of SLA on a mailserver? Emails get retried a number of times before they are considered undeliverable.
SMTP requires sending servers to retry for at least 4 days, so 98.9% uptime over a year is guaranteed to be sufficient so as to not lose any mail ... so that is "several" as in "one", I suppose?
Of course you can have a lot more downtime than that, even just 20% uptime would be sufficient if timed right to ensure that no mail gets lost.
Or in other words: My central heating system has higher availability requirements than my mailserver, so I guess everyone in non-tropical places has a non-trivial endeavour going?
The site has a link to the starttls everywhere site right at the top - but it's also worth mentioning that the regular 'certbot' client for https everywhere can also be used to create certificates for your own smtpd. The easiest way is just having the sudo ability to temporarily open port 80 for the challenge/response process.
Thanks for the plug. The same developer at EFF is working on both STARTTLS Everywhere and Certbot Postfix integration. :-)
(Anyway, the most significant difference between the two is that STARTTLS Everywhere can also enforce certificate validation on recipient sites that request it, which Postfix won't otherwise do.)
> The easiest way is just having the sudo ability to temporarily open port 80 for the challenge/response process.
The easiest way is to forget about HTTP and use DNS verification. Just create a dynamically updateable TXT record in your zone that the LE client can update and then point CNAMEs for the names that you want to create certificates for to that TXT record.
I am too paranoid and untrusting to give the LE automated tool the ability to edit DNS zonefiles. My DNS setup is quite far removed logically from the mail server setup, intentionally.
You don't need to "edit zonefiles", and whether it's "logically removed" is kinda irrelevant?
You simply set up a dynamically updateable zone and then have the LE client call nsupdate, it couldn't be any easier, you don't even need sudo.
You could also delegate DNS for your MX to the MX itself, of course, then you'd have pretty much the same setup as with the HTTP server on the MX, just using a DNS server instead. But that's just too complicated, really, using dynamic DNS is flexible and allows you to easily add other services that need certificates on any machine whatsoever with no need to have an HTTP (or any other) server running on the machine reachable from the outside.
So I support more encryption everywhere, but this is acting like there's something new? I know I've been using STARTLS on SMTP for almost 20 years… Am I missing something?
The "STARTTLS Everywhere" project [0] -- the one thing this entire article is talking about -- is what's new.
> ... I've been using STARTLS on SMTP for almost 20 years ...
That's great -- I have too -- but I have some bad news for you: not everybody else has been! (I know, right!?)
The EFF is simply trying to bang the drums to get everyone's attention and then convince those that aren't already using STARTTLS (everywhere, hence the name) to consider doing so.
The good news, for you, is that you're already done. There's nothing for you to do, unless you want to add your domains to the "policy list" so that others who are participating won't send mail to you unless encryption is in use.
Fortunately, the EFF has already written about this extensively [0], so there's no need for me to rehash all the details (coincidentally, that same exact web page is also reachable by clicking on the first link in this article).
I submitted my domain and it’s not on the list, and there has been no communication about why or why not (given that it passes their security checker and requirements).
That would be fine, too, and reflects in part what MTA-STS is trying to do.
A point that may be obvious, but I would like to make explicit, is that unlike on web, end-users have no good way to express whether they prefer security over deliverability on a connection, which makes downgrade attacks more difficult to deal with. Everyone defaults to preferring deliverablity (falling back to plaintext). In the absence of user intent, it's more important for an email sender to know the recipient mailserver's intent: what level of security should they expect? This can be achieved at varying degrees through DANE, MTA-STS, or a "HSTS Preload List" equivalent for email, but there is currently no reliably deployed standard for this.
(edit: something I forgot to mention: of course, you'd have to trust the initial DNS lookup as well)
What would be more/also interesting would be a way of expressing this intent by a sender.
Seems like (with absolutely no thought here) a new header could me minted for this - eg. "Transport-Security: Require". Tough part as usual would be getting MTA and client support, but switching to a per message approach would at least allow incremental rollout - and critically I think - allow introspection by following bounces.
I have my mail server require a TLS connection to a host before sending mail, but perhaps 2% of the time I get my mail bounced by the target server because it doesn't support it, including my old university. It's really crazy that a 20+ year old, standard technology is not literally everywhere, especially in big organisations.
Incidentally, that same university in the past year started injecting footers into people's emails, so I guess their primary focus is neither privacy nor security.
At present not allowing plaintext port 25 will break a lot of email functionality, there are a ton of old and poorly set up mx out there. I have my postfix setup as "may" for TLS, not required.
But for actively maintained servers, could they not be configured to support TLS for clients that request it without breaking plaintext email for old clients? I'm trying to understand whether supporting old clients really means not supporting encryption features for up-to-date clients.
I was referring to server-to-server mail exchange.
Yes, this is how my server is. When exchanging smtp with another domain's mx, if that other smtpd will not do TLS, it sends plaintext mail. For clients submitting mail via smtp with auth, such as thunderbird on my desktop PC, of course it does TLS1.2.
Between this and SMS, the NSA (et al) had a gold mine of data when they directly tapped the lines.
I remember reading they were really upset when this email providers started using this by default... always a good sign when warrantless/dragnet surveillance is made harder.
> The STARTTLS Everywhere project has created a “STARTTLS Policy List” where supporting organizations can assert that they always use STARTTLS using a valid certificate. Organizations should join that list and check that list before they send email.
We don't need a new separate place to publish the TLS ability of our mx. This can be done with a txt record in the DNS zonefiles same as spf.
It would need some sort of consistent rfc-guided format for what TLS policy re: downgrades, port 25 plaintext, etc, that you want to publish.
Not too dissimilar from publishing an HSTS policy, but done by the authoritative nameservers for a zone rather than on an individual httpd.
Because it's not backwards compatible, and would prevent delivery of a large amount of email.
We just need a way for domain owners to signal that their domains definitely accept email over STARTTLS. At that point, senders which recognise those signals can start to enforce that mail which they send to those domains MUST use STARTTLS. Fortunately, those signalling methods are beginning to rise to the surface. We have MTA-STS, and now also a preload list thanks to StartTLS Everywhere.
That is completely insecure. A MITM would just block access to the tls-on-connect port, forcing the sending host to retry on port 25, where they would strip the STARTTLS advertisement.
Unless you're recommending to not fall back to port 25 after a failed connection to the tls-on-connect port. In which case, see the comment you were replying to where I said it would be backwards incompatible.
Both of the solutions I mentioned are backwards compatible, and protected from a similar downgrade situation.
Right, I should have said that I was recommending not ever sending on 25. The STARTTLS mitm is just too easy for large attackers.
IMHO, being backwards compatible is necessary for a transition period. The messaging to SMTP operators should be... get on board, or you are going to start losing mail in 1 year.
Even Chrome is going to show all http as insecure in the very near future. Which reminds me... I need to get all my sites upgraded.
The problem that prevents such a standard from ever taking off is that, unlike TLS, there is zero interest from major providers to push end-to-end encyption, because they absolutely need access to the plain text for commercial reasons. So they advertise STARTTLS as "email encryption". By that token, CDMA and GSM are "email encryption" as well, it's not like they push the plain text naked over radio.