Hacker News new | past | comments | ask | show | jobs | submit login
JMAP: Like IMAP but Not Really (unencumberedbyfacts.com)
325 points by jasonmunro 3 months ago | hide | past | web | favorite | 228 comments



FastMail is behind this protocol and from what I've read JMAP has evolved out of their web interface.

I've been a happy customer, even though lately I flirted with going back to GSuite for my personal email, but after a trial realized that Gmail does many things well, except for being a good email service. So I went back to FastMail and renewed for another 2 years.

Seeing this new protocol is exciting, because JMAP is being standardized at IETF. A breath of fresh air to see a new standard being developed.

Also from what I understand JMAP should be friendly for mobile usage. They kept notifications out of it, you're supposed to implement notifications using whatever the mobile platform provides. Interacting via JMAP is via plain HTTP requests, which is super cool.

I can totally see myself implementing a simple email client for automating online services. For example if you implement a commenting system for a website, you might want to do replies by email. That would be a cool project for me to try out.

I wonder if FastMail exposes JMAP publicly yet. Haven't seen any mentions in their admin or docs thus far.


I've been a happy Fastmail customer too, until I was made aware that you can impersonate other Fastmail customers by just spoofing the email address. Their servers just happily accept it. SPF and DKIM all pass with flying colours, and the only way you'd know it's happened is if you have DMARC on and happen to notice a pass in the report you don't remember sending. Well, that is if the recipient doesn't reply to the spoofed message - hope the damage wasn't already done though. It's effectively impossible for the recipient to know it's been spoofed.

The worst part is I think Fastmail is aware of it and just don't care (believe that's why they mark their emails with a green tick and text). I understand that email has never been really authenticated, but this just throws any trust I had in Fastmail out the window.

I will be evaluating other mail hosts at the end of my subscription.


SPF/DKIM are designed to prevent spam and not to prevent spoofing of From. There is a difference.

SPF has nothing to do with the From header. And the DKIM signature does not have to match the sender’s domain, the signature can be that of any domain. This means that for practical purposes, anybody can send spoofed emails. That an email is signed with DKIM, that doesn’t mean much and it is meant to build a web of trust between servers, but otherwise it is useless for the users themselves.

They wrote a blog post about how SPF/DKIM work: https://fastmail.blog/2016/12/24/spf-dkim-dmarc/

If you want to let people know which emails are from you, the From address is very weak. This is because the From/To headers tell you nothing about the source and the destination of the message, according to the email standard. Read that blog post for details.

You need a proper signature via PGP or S/MIME if you want to ensure that the receiver knows the message is from you. And unfortunately this requires education and email clients with support for such signatures (most desktop clients do), but that’s email for you.


Sure, I get that. I get there is a whole weird and wacky world of email use that is considered legitimate and needs to work that way for a myriad of reasons. I don't get why their MTA cannot at least have an option to reject mail from your domains if it's not being sent using your account credentials.

The average layperson will not get that. I'm fairly sure if my mother received an email that wasn't delivered to a their spam folder saying "Hey, remember that old copy of my birth certificate you have floating around? Could you send that. Also, CC my good friend bad_user@fastmail.com" that she would call me first - if I was reachable. Also is totally ignorant of digital signatures and most likely unable to verify any present anyway.

As much as I dislike Google and try to avoid their products and services at all cost, at least I have confidence this wouldn't happen with them. Not that I would go back, but it's still concerning.


But that’s the point, you can send a spoofed email to your mother that will not go into her spam folder, even if she uses Gmail.

The only way Google could protect you is if the From address is from @gmail.com (maybe, not completely sure). But if you have your own domain, you can’t have that protection. Sure, you might not be able to use Google’s own servers to send that email, but email is federated so you can use somebody else’s servers.

The only thing that stops spammers from doing more of this is the web of trust happening between email services. This is precisely why if you setup your own server, you’ll start off with a negative reputation and your emails will end up tagged as spam depending on the destination.


> But that’s the point [...]

No, that's not the point.

> Sure, you might not be able to use Google’s own servers to send that email

That is the point. Why does Fastmail allow this where Google doesn't. At best, it's ignorant and intentionally misleading. At worst, downright malicious and ripe for abuse.


There's much I don't know about email, so take what I say with a grain of salt. I imagine that maybe this could simply be a low priority issue for Fastmail because such a restriction would not be a protection for their customers but rather a restriction/disservice to them to potentially protect everyone else.

I also wonder if there are superusers that have a legitimate use for sending emails that have a different "From".

Something to think about is that, looking at the postal mail it was designed after, I don't imagine a postal office would reject me if I tried to drop off mail authored by someone else. They don't check the "From" in the envelope with my ID or anything. In fact, many envelopes don't even have a "From", and you don't even have to face a human when dropping off your mail. All the postal office does is provide access to the global delivery network for a fee.

It might be more apt to think of email providers likewise as network providers that allow transparent access to the global MTA network.

Both postal and electronic mail rely on signatures for proper authentication. It's only that electronic mail's (cryptographic) signatures are more secure but more difficult to use by laymen.

Maybe this issue ought to be thought of a similar to how illiterate people sign paper documents by making an "X". I imagine it's trivially easy to spoof documents supposedly signed by them, and even mail them. I wouldn't blame the postal office for accepting such spoofed documents.

Computers being relatively new and all, perhaps it isn't that bad to think that most of the world is still computer illiterate even if they think otherwise because of their ability to use point-and-click interfaces designed to be used even by illiterate young children.

What I think is needed is better computer education.

As to where this expectation for "From" to be validated comes from, I imagine it's something we've grown accustomed to from our use of centralized services. It would be really bad if a message on Facebook or Twitter could be spoofed, but those services are centralized, so restricting their users equates to properly protecting their users. Email, however, is decentralized. That's a good thing, and the proper way to do authentication in an decentralized service without making it more centralized can only be by non-spoofable signatures and not by trusting validations from independent service providers.


FWIW you can inbox mails from spoofed @gmail addresses on gmail.


At this point, none of the mainstream email websites or applications even show the From address, just the name that came on the email, or what matches from your address book. So, I fully expect people to get mail from webmaster@johnssportsupplyemporium.example.org with a From name vaguely related to me and assume I sent it.


You should just send an email impersonating their CEO to their IT telling them to fix it ;)


That will definitely work


I'm sad to hear that but it seems to be true. I understand they have legacy stuff to support but it seems negligent for a company that exclusively deals in email to ignore something like this.

https://www.fastmail.com/help/technical/senderauthentication...


This!!

I reported the same problem to posteo.de in March 2016 and still have not received a satisfactory answer, though it seems they have some counter-measures in their webmailer nowadays. The fun part was that as a "no logs" privacy-oriented provider, they were not even able to track who sent them a complaint from their own support address ¯\\_(ツ)_/¯

As a comparison: at disroot.org I found the same problem, and it took them a few hours to repair their postfix configs.


Is there anything in the email headers that shows the authenticated user? My preference would be that email providers rewrite the From: header to match the authenticated user and that the sender uses the Reply-To: header to direct replies to a destination of their choice.


Not good enough, as many email clients show both of these fields. The whole idea why people use this is to send and receive with one address only, even if you've authenticated with another.


I believe most email clients will default to using the Reply-To: header to determine the value of the To: header in the reply, so if both headers are specified and even if the email client displays both of them, then the recipient would have a chance to determine whether the email is authentic or not.


Fastmail includes a header - decodable only by Fastmail - that can identify the sender account. And IP address if sent through SMTP.


Do you have any reference for this issue?


This freaks me out too, and it turns out it's true: here's a mention on their Bug Bounty page [1]:

"Email spoofing bugs do not qualify. We are quite aware that users can set arbitrary From addresses on emails, that our SPF records allow arbitrary hosts to send email as our domains, and that our DMARC policy is not enforcing passes. These policy decisions are by design, and we track the actual sender in a separate header."

[1] https://www.fastmail.com/about/bugbounty.html


It's not entirely clear to me what they could do about it. Since they are an email provider, they probably don't have control over the networks their customers send email from, and what else their customers do with their domains.

Someone could decide to forward their other mail to their fastmail account. Should they then potentially risk email their other customers send to that address? DMARC headers tries to solve this, but the world is dirty, mailing list software suck, and their they would have to take the blame for problems outside their control.

I can understand the decision. They could probably do something to show good intentions, like flagging suspicious email and making sure their own email software shows appropriate warnings, but it's never going to be perfect.


They could configure their MTA to rewrite the From: header so that the value matches the authenticated user and that the sender users the Reply-To: header to redirect replies instead.


They need to provide the ability to use SMTP servers other than their own for @fastmail.com users.

SPF, DKIM and DMARC do not provide authentication of non-envelope headers like From: and To: etc, unless they are specifically included, but there is no way to publish that you require those headers as part of the DKIM signature.


Exactly. This is also what makes SPF and friends a bit of a pointless exercise. Even if they had global unanimous support end users don't really care about the envelope from anyway.

Stopping phishing is hard. End users mostly are fooled by a little padlock in their web browser, and that's a much simpler trust model. Eliminating email dressed up as web pages would probably do more to combat that than authenticated sender models ever will, but nobody really wants that.


I think the thing that is concerning to me is not so much that users don't care about the envelope from, so much as it is that other email providers' anti-spam measures may block my email if some spammer start spoofing me. Then, poof! I can't email any gmail accounts anymore.


Use GPG if you're concerned about authenticity. It's the only way. What you describe is not a problem with Fastmail, it's a problem with basic communication without cryptography.

The problem is if any email service did this you'd start trusting the "from" field and that is wrong. Do not trust the from field. It's as simple as that.


What is the reason for allowing this? Laziness?


They would need to make some tie between sasl authentication and what FROM: headers you're allowed to use. I don't know what MTA they use, but there are MTAs that have that feature. It's controlled_envelope_senders in postfix. I assume other MTAs have similar features.

It sounds kind of lazy to me. Though I'm sure they would get lots of complaints if they turned it on...some mailing list software depends on spoofing, for example. Or web based "contact us" forms. So perhaps it's just to avoid lots of support tickets.


The reason is probably that nothing can stop the successful spoofing of the From header. DKIM is a signature for authenticating a domain, however that domain does not have to match the domain in the From header.

Take a look in Gmail at a signed email and you’ll see a “Signed by” field in its header info, with a domain name as a value.

Also the SPF setting has nothing to do with the From header either.

In other words the “From” value cannot be protected, unless you sign your email with PGP or S/MIME.


That's tripe.

They know who authenticated to the SMTP server, so they could enforce that the From address is who it was authenticated by. Otherwise, they basically act as an open relay.


Sending from multiple From addresses is a common use case. I send from at least 4 different email addresses all hosted by fastmail in the same account. Having to create different logins to authenticate each sender would be a huge pain.

Plus it's not a unique problem to fastmail.


Gmail requires that you prove ownership of an email address by clicking a link in an email before letting you choose it as a From: address. I think this is a good compromise.


You can also take a blacklisting approach, where it's open-by-default and users shall be able to restrict any domain to properly authenticated users only. That way, it is purely a security enhancement for those who want it (like me).

I demonstrated this behavior to eggsampler after discovering it quite a long time ago by messing around with HTTP payloads in their web interface - it's wild to me that FastMail will use the DKIM private keys from an entirely different FM account to sign your messages.

Unlike eggsampler, I won't be ditching them, but I hope that FM reconsider their policy eventually. That they have awarded themselves the privilege of a "green tick" on their own official emails while throwing everybody else to the wolves is slightly ironic.


Presumably they could require that the from address is one your authenticated user is allowed to use, right?


I'm not certain, maybe there's a technical reason they can validate account credentials but not map credentials to addresses/aliases. Doesn't instill confidence either way.


Just to make it clear, is this true if you use your own domain, or just if you use any of Fastmail's domains?


Seems to be just Fastmail domains but there might be some setup involved for custom domains. See their docs

https://www.fastmail.com/help/technical/senderauthentication...


If I understand this correctly, you need to be managing your own DNS (not letting Fastmail do it), and you probably need to set p=reject in your DMARC so that non-fastmail servers can't spoof your addresses.

And if Fastmail allows Fastmail user A to spoof Fastmail user B, then the above still only protects you against non-Fastmail customers.


I have received a spoofed email from my own domain, so I believe it's not just Fastmail domains but any custom domain on an account.


Do you host your custom domain with Fastmail? I may have misread but the article seemed to suggest you could potentially setup dmarc via the DNS page or whoever you host with.


The From header has always been spoofable. It's just most ISPs (and Google) chose to disallow it to address low-hanging fruit in the fight against spam.

But anyone can set up their own postfix/qmail/sendmail server and put anything they want as the From.

Or am I misunderstanding the issue here?


The difference is that using another fastmail account to spoof someone@fastmail.com will make the email look much more legitimate (DKIM and SPF wise) to other servers than when it comes from your random.emailserver.domain.foo.


...wow. Do you have any recommendations for alternatives?


> after a trial realized that Gmail does many things well, except for being a good email service.

This.

I remember back when Gmail was new and hot. It was unlike any other email service out there, and ridiculing people for using inferior email-solutions could to a certain extent be justified.

While other webmails were slow, had constantly reloading pages and what not, Gmail was fast. It was amazingly fast. Gone where the 5 minutes making the webmail work for you. You just sent the email and you were done. Just like that. Back then, this was unheard of.

These days though? Everyone is still pretending like GMail is the only game in town, when I think they have one of the worst webmail-interfaces out there. And it's slow. Oh god it is slow. And god forbid you try to load it in a browser not Chrome, because then it just grinds to a complete halt.

So yeah. Happy Firefox-using FastMail-customer here. You couldn't get me back to GMail even if you paid me.


The main thing gmail offered back then was a frickin GB of email storage. (Which spawned a host of other things, like a FUSE filesystem backed by Gmail.) No other free provider was even close! Hotmail offered email measured in 10s of MB and I'm sure Yahoo was similar.

(I got my first gmail account in 2004.)

Yeah, the UI wasn't bad, but there was less email back then and I didn't (still don't) really mind being served HTML repeatedly instead of some "AJAX" application, which is what I remember the technique being called. (It used to be the latency to the server was poor and JS hid that; now, on 2-3ms fiber, it seems that the JS actually introduces a lot more latency than it hides.)


Hotmail provided just 2MB of space when gmail launched on 01 Apr 2004. They increased it to 250MB in June 2004.

source: https://www.pcworld.com/article/116657/article.html


I used Yahoo Mail at the time, which had a limit of 6MB (and IIRC that had been an increase at some point). Soon after GMail came out they upped that to 250MB. Searching now I find an article saying this happened December 2004. I seem to remember it happening closer to the GMail announcement though but I could be wrong.


Does FastMail let you have complex conditions in filters though? One huge value of Gmail for me is that I can have arbitrarily complex nested conditions to filter mail using a formula. I've been looking for even more flexible filtering (e.g. custom scripts) and yet I have not even come across anything on par with it.


fastmail lets you write complete scripts for filtering using the Sieve programming language. https://www.fastmail.com/help/technical/sieve.html


Sieve is great. I highly recommend it over, say, procmail, or Outlook filters (especially Outlook Web Access).


Isn't procmail dead?


What does dead mean to you? It still compiles and you can install it from various binary repositories.


It's not even maintained and the official site seems long gone. How do you see this as alive?


Holy cow, I will have to look into this. Thanks a ton!!


filters cannot be applied retroactively, however.

be sure to write your filters before receiving anything. Especially important if you're importing your email for the first time... and insanely annoying if you want to debug such an advanced filter.

as always with fastmail: It has great features... but always with a massive caveat.


Ooooof, wow, thanks for the heads up. There's no way that's going to work... I edit my filters and apply them retroactively... There's no way I can set them in stone the first time. :( is there any email service that doesn't have this limitation?


Its not on FastMail, its on Sieve. It is a hook that runs on receiving. To my knowledge if you want Sieve this is the only option. I use it with Protonmail and it is the same way. I also run it on my own mailserver.

One nice thing, for me at least, is it runs prior to the email ending up in your inbox and I am uses to Outlook rules were rules execute post arrival.


Sieve is an RFC email language. There's no reason Sieve recipes couldn't be applied to a corpus of existing email; that's an implementation detail of FastMail. Sure, it isn't necessarily the same as pre-receive (for example, anything you discarded the first time isn't going to show up again, and any manipulations made the first time aren't going to undo themselves). But it could be done; it's not really Sieve's problem.


Wouldn't using something like https://github.com/lefcha/imapfilter also be an option?

It would be awesome if some day there was a "jmapfilter". I think JMAP would be really efficient for this use case.


Most of my filters are simple searches. And if you can search, you can apply it retroactively.

Also FastMail’s search works for any email header so it can be more potent than that of Gmail.

I wonder why can’t they add a checkbox to apply the rules retroactively. Seems to be simple enough.


I considered FM a while back but... you pay per account! $50/account no less! I separate my email into three accounts across two domains and my wife does the same and I have kids who'll have their own email addresses at some point.

I can't even begin to consider using it at the price point I presume I'd be in.


I use a single account for 5 domains, with multiple aliases per domain.

For IMAP folders you can set a default identity and email in a folder stays in that folder without cluttering Archive, like in Gmail. So in effect you can do much with a single account.


You can have all the domains under one fastmail account per user (I have ~8 domains, and ~20 users). But yes, you still have to pay for each person


Check out migadu.com; They price their plans according to how many emails you anticipate to send.


$30 for a smaller inbox though. I have all my email since 90s in there and it all still fits just fine.


For people reading along: $50/yr gets you 25 GB; $30/yr gets you 2 GB. I'd guess 2 GB would work just fine for most people.

OTOH, my relatively new gmail account has >4 GB in it. Other accounts have 0.8 and >3 GB; none of these were used for file storage (FUSE) or anything like that.

https://www.fastmail.com/pricing/


I wonder, maybe I then just don't receive that much email? I did a quick count in the folders view, there's currently about 35k emails which take up about 1.7G of space.


If I read their pricing page correctly, you have to be on the $50 plan to use your own domain name.


> Also from what I understand JMAP should be friendly for mobile usage.

I read the FM blogs and I remember them mentioning that low battery consumption was one of the design goals.

The JMAP site [0] has:

> JMAP is designed to make efficient use of limited network resources. Multiple API calls may be batched in a single request to the server, reducing round trips and improving battery life on mobile devices.

[0]: https://jmap.io/spec-core.html


> Also from what I understand JMAP should be friendly for mobile usage. They kept notifications out of it, you're supposed to implement notifications using whatever the mobile platform provides. Interacting via JMAP is via plain HTTP requests, which is super cool.

Sounds actually not cool. Is there really no standardized way to do notifications? The world consists not only of shitty mobile walled gardens which insist on or strongly favor their proprietary implementations.


Using what is provided by mobile operating systems is the standard. FastMail is decoupling JMAP from notifications because it's a sound architectural decision for that reason.


If you want good battery life, there is no alternative to the mobile notifications provided by the platform, because otherwise you have to always keep your own connection open to your server.


please don't spread misinformation - abusing long live socket connections is what drain the battery - having one open and managing it in sane manner is quite battery efficient. Google brainwashed everyone to force them to use their push service, which is just plain awful... (and requires open connection... sic!)


That's simply not true if implemented correctly. Maintaining an open connection yourself has at most minimal impact on battery life. See for example Conversations (XMPP client) which is able to do this just fine on Android.


> ...Gmail does many things well, except for being a good email service.

Gmail/Gsuite seem to be good email services from my perspective as lay user and occasional admin. Can you expand on why you think they are not good email services?


Here's a few things:

- constantly changing things around. UI gets less intuitive for every release.

- slower for every release

- violates IMAP standard by re-using IDs across tags/IMAP folders, risking actual data loss. Example: if you in a real email-client try to delete a email from a single folder, you will also delete this email from all other folders it has been "tagged" in.

- similar IMAP issues with sent emails. Hard to track sent emails from email-client, unless you allow the email-client to explicitly save a copy to sent-emails folder. But then you suddenly have duplicate emails in the web-UI.

- Makes an open standard (internet email) proprietary.

- Hell to integrate with: See above.

I'm sure I could go on, but really. If you still consider GMail best in class, the only possible explanation is that you haven't seen anything else.


The messed up deletion behavior baffles me to this day. The stock mail app on iOS will behave correctly if you muck with the Gmail POP/IMAP settings and turn off 'Archive' in MobileMail, but I have zero confidence that this will work on macOS because Apple pays less attention to the desktop and Gmail don't give a crap if actual client applications work correctly.

They just assume everyone uses the web interface.


> They just assume everyone uses the web interface.

More likely, they want you using their app and their web interface, and merely tolerate third party apps. For now.

Google wants you in the Googleverse forever.


I’m not a fan of Gmail either, mostly because it lacks push vis iOS Mail (which Fastmail supports, so it is possible), and its non-standard tagging system doesn’t not play well with any third-party client that does not take extraordinary steps to accommodate it. Folders are better. Its proprietary apps are terrible and its web interface is weird, unintuitive, ugly, and slow.


I find it's "non standard" tagging system is a massively superior model for managing email. I really wish there were alternatives that did similar, and a protocol like IMAP or JMAP that assumed the tagging based model underneath.


>JMAP is a REST API so it uses HTTP requests and responses to issue commands and get the results. Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”.

Describing this as REST is really strange. Defining your own operations over an HTTP POST is what SOAP and other RPC style web services do and specifically what REST isn't. But I guess that a lack of a standard behind REST ended up with the term being used for everything.


REST is what Roy Fielding wrote about in his thesis. It is an architectural style, not a protocol.

It involves two endpoints exchanging the state of a shared resource. It needs to be compliant with the constraints of that style.

People think that REST must be over HTTP, but it can be over any protocol. The essence is that it is a style of systems design, so JMAP can be considered RESTful as described in the link above.

REST is one of the real patterns in software architecture, a set of constraints, not a set of structural elements.


I know all this, but this API doesn't meet that definition. As cruegge has linked the JMAP developers themselves don't consider their API REST. From the description it seems more like an RPC that's transaction oriented and for very good reasons.

The fact that this is a discussion though makes my point. By REST being only an achitecture style with very loose definitions makes it arguably fit for all different kinds of APIs which in turn has made the term useless over time. Maybe we could use a set of technical standards (like SOAP) for different common API implementation solutions within the bigger REST idea. Discussions like REST vs SOAP are like comparing OO and Haskell, one is a concept/pattern, another is a specific technology.


> As cruegge has linked the JMAP developers themselves don't consider their API REST.

That wasn't my takeaway. They don't consider it HTTP-based REST, but they do consider it RESTful. The FAQ's question is from the perspective of someone who doesn't make that distinction.

You can model things RESTfully, but the encoding of REST into a carrier protocol (commonly HTTP, but it doesn't have to be) is a separate matter. The "very loose definitions" largely stem from the encoding, not the modeling.


It's ambiguous indeed which again reinforces my point. The fact that you can issue several commands at once to be executed in order is strange for REST. And the REST definition also includes using common operations (GET/POST/PUT/etc if using HTTP) and this has invented it's own set that's specific to only this one API. So my read is that they're saying "we're actually more REST than a lot of things around but don't consider ourselves that".

>The "very loose definitions" largely stem from the encoding, not the modeling.

Sure, but the fact that we don't have strict definitions for common encodings means that you then have a discussion on it every time. It would be nice to have Standard A, that defines a common way to encode a common type of REST API (HTTP/JSON/etc) and we can just say our software uses that and REST is implied. Instead we get a situation where JSON-over-HTTP is perceived as being REST when it actually fails as a test in both ways. Some things are REST and don't use JSON or HTTP and some things are not REST and still use JSON and HTTP.


This is entirely fair -- I agree! REST isn't a protocol in the same sense that HTTP is. Although I believe a protocol's semantics and encodings should be distinguished a la REST, a way to talk about a particular encoding would be a very nice thing to have.


REST doesn't really have anything to do with the HTTP methods you use. It's about handing off state between the client and server using stateless requests instead of maintaining a persistent, stateful session. This blog post is a good overview: https://programmingisterrible.com/post/181841346708/what-the...


They have an FAQ item concerning that: https://jmap.io/#why-is-it-not-rest-based?


At this point I guess any API that runs on top of HTTP is REST? Strange world.


I guess REST became catch phrase for all JSON over HTTP protocols.


I believe that one could make a REST API over other protocols like IMAP or NNTP.


Maybe unpopular opinion, but i'd take this any day over a "proper" REST api requiring DELETE and PUT.

Also i'd classify a REST api as anything that does http requests and consumes proper JSON. As long as this is true , the rest is squabbling :)


> i'd take this any day over a "proper" REST api requiring DELETE and PUT.

REST doesn't need to use any HTTP verbs other than GET and POST.


REST doesn't need to use HTTP.


As rswail said, REST doesn't need to use HTTP at all, however, if it does then it "should" use it fully including using HTTP headers for metadata. In theory, of course, you don't need anything other than POST


I really want to see some innovation in the email space. The landscape is like a sea of false promises and dashed dreams. SMTP is one of the bread-and-butters of the internet, yet it just doesn't seem to be moving forward (maybe that's for the best), and no one's building extensions on top of it.

Maybe I'm naive in thinking it was possible but we could have avoided this whole "make an account on X messenger so we can talk", if someone had just jammed XMPP and SMTP together.

Every few years I'm tempted to try and write a SMTP server (MX) myself but then realize that postfix has been around so long and is the go to choice for a reason. I don't have 20 years of figuring out how X mail server interprets SMTP to be as reliable as postfix. I settled for trying to wrap it[0].

[0]: https://gitlab.com/postmgr/postmgr


Yes, postfix has been around for so long and it is very solid. However, to run a mail server nowadays, it still needs a lot of configuration work and even additional daemons to add SPF checks, forwarding with SRS, and DKIM signatures. All of this is possible to be added though, which speaks for the flexibility of the postfix design.


This is exactly what I was hoping to add to it with my postmgr project. There's projects like Mailu[0] which are fantastic as well.

[0]: https://github.com/Mailu/mailu


> if someone had just jammed XMPP and SMTP together.

Xmpp is/supports federated/ion (user1@server1.com can message user2@server2.com).

But the major players (by numbers) wanted silos: Google (talk) and fasebook (first gen of messenger).

They basically went lol, screw users (arguably because: spam. But hello, Gmail? And today fb spam...).

So thank Google and Facebook for deliberately gimping it so you can't fb message user@gmail.com or gtalk first name.lastname123@facebook.com.


The official reason for Google to give up on this is that there was a real problem of incoming spam from lousy domains, and no major player wanted to play the federation game with them. It's not as simple as you make it sound.


1) they are able (forced to) handle spam for Gmail (and internally for gtalk) - I'm not convinced by the spam argument.

2) there were only two "big" players, Google and Facebook. The benefit of federation would be as with email: an open internet with federation across organisation level services (community/company run servers).

The fact that Google didn't implement ssl for federation doesn't mean people "wouldn't" federate with them, it mean Google didn't make a real effort.


Email is "good enough".

The real hole is "instant messaging". Somehow every few years we got from ICQ to AIM to Paltalk to Skype to Facebook Messenger to Slack to ... (at least Altassian had the decency to take HipChat behind the woodshed and shoot it)

The strange thing is that these services dry up, blow away, get replaced, but they don't seem to improve on what came before.

There is a standard, XMPP, but the only people who care about it are firefighters, cops, and soldiers. For all the anger at internet giants these days, I can't see for the life of me why people aren't pushing for open instant messenging.


XMPP’s problem isn’t that people don’t care about it. Its problem is a double whammy of having zero branding/mindspace presence and being too technical for many users to bother with.

The second point is especially big. People expect to be pointed to a singular app/site, not “choose whichever client you like”. Your average Joe just doesn’t conceptualize services as being separate from clients in the first place, and even if they did having to choose a client is a dead end.


I totally agree with that. I've good experiences recommending Quicksy[1][2] though. It makes not only account creation easy as pie, but also helps people finding their peers by their phone numbers.

Give it to your average Joes and Janes and they will just use it like any other messenger. But since it is pure XMPP, can just use any other Jabber account to chat with them. :)

I recommended it a lot the last weeks and most of my pals just use it without any hassle. At least in my bubble most of the people have besides WhatsApp also Telegram or Viber or WhatEver installed. So it's for them Quicksy is just yet another app. But this time one, which helps to get out of the walled garden. :)

[1] https://play.google.com/store/apps/details?id=im.quicksy.cli... [2] https://quicksy.im


Well, there's [Matrix](https://matrix.org) in this space which I would characterize as pushing.


I didn't explain it fully, but the alternate future I was imagining was one where SMTP could evolve/instantiate a connection for "ASAP email" -- AKA instant messaging. Kinda like how HTTP can upgrade to websockets.

There would obviously be a lot to work out, but if something like this was in the SMTP standard (or at least introduced), I think eventually email providers would race to differentiate themselves with support for it, and we might not be where we are today. In my view there's no reason to even require the SMTP servers to serve chat traffic -- as long as they could hand off reasonably, and everything spoke the language as specified in the spec (or at least came close).


I don't want instant messaging in my email though. If you need synchronous communications, use a chat client.


This is probably where we differ -- I don't think limiting email to asynchronous communications helps anyone, and it could have been written in an extensible manner to also support synchronous communication.

Even if I'm completely wrong and email should be left asynchronous, SMTP could at least have introduced some standard around chat negotiation and a protocol negotiation process for remote chat servers/agents.


> Even if I'm completely wrong and email should be left asynchronous, SMTP could at least have introduced some standard around chat negotiation and a protocol negotiation process for remote chat servers/agents.

Wouldn't this be best handled with SRV DNS records or some other service discovery process outside of SMTP?

https://wiki.xmpp.org/web/SRV_Records


I agree that it would be but my wanting to bundle it with SMTP is for forcing the linking of the two concepts -- I guess my argument is less technical and more that I think the two concepts should have been linked early on -- I think if we'd bundled email and instant messaging (aka "live" or "instant" email), we would have encouraged service providers to offer the service in a standards compliant way.

I don't have a preference on exactly how the service discovery would work -- I'm merely suggesting instant messaging could/should have been an extension of the SMTP spec like websockets were an extension of HTTP.

[EDIT] - Looking over the link you sent, that is exactly what would have made a good addition in my mind -- if SMTP had some sort of functionality to suggest that XMPP was available and where to check (DNS SRV records)


After thinking about it some more, I agree with you. There should be some command or extension in SMTP and JMAP that, if the user permits it, the availability of XMPP could be exposed and a redirect of some sort could be provided (similar to intentions on iOS).


IRC works pretty well. I really fail to understand why any tech project puts their chat on Slack/Discord/Whatever when IRC is easy and open.


First, IRC isn't easy for newbies. I tried to adopt it at the first tech company I worked for, and our support folks had a lot of trouble getting the basics down. There could be a client that makes things easier, but I've never seen one as easy as Slack.

Second, IRC alone doesn't provide a bunch of features that everyone expects nowadays. You have to host or pay for a bouncer if you want to see what was said when you were offline, for example. Gotta use a 3rd party service for push notifications on iOS. Again, there's no reason why this couldn't exist, but it's another product, not just IRC.


> IRC isn't easy for newbies. I tried to adopt it at the first tech company I worked for, and our support folks had a lot of trouble getting the basics down. There could be a client that makes things easier, but I've never seen one as easy as Slack.

Years ago, I started out with mIRC on Windows 95. I didn't know many IRC commands, but as I recall, you could navigate through the menus to do things like list channels, join them, part from them, etc. So, I don't think that an application like that should be any more difficult to use compared to Slack.


"if someone had just jammed XMPP and SMTP together" - wasn't that kind of what Google Wave was aiming for?


Yes. And the vision for Google Wave was great. The execution was just abominably bad (primarily on the frontend/UI side), and Google's failures in execution carried over reputationally to the—now dead—Apache incubator.


Not just the UI side. The underlying protocol was also pointlessly over-engineered. SMTP is still around because it is simple. You can get a basic SMTP client running in an hour or two. You can get a basic server running in a similar amount of time, at least in a higher level language. That matters, not because anyone will have anything usable in that time, but because it lets you start to experiment very, very easily.

Wave suffered from a UI that was a mess layered on top of a protocol that made it really hard for people to get started on experimenting with alternative frontends, and without a user-base giving people a reason to persist figuring out how to interoperate with it. Had the protocol been simpler, the UI mess might not have mattered so much - people might have come up with their own ideas.


The S in SMTP literally stands for Simple. And it was developed in 1982. So it was considered simple in 1982.

That gives you some idea how simple it is.

https://tools.ietf.org/html/rfc821


It is simple today, except for the authentication.

Of course, if the authentication doesn't work, nothing works!


You don't even need a client - connecting with telnet and sending emails is pretty straightforward.

I guess the modern equivalent is checking if a REST API is usable just with curl.


Further than that, it seems to me SMTP is functionally included inside XMPP so there's no need in mixing them. Perhaps having an SMTP gateway that feeds into an XMPP server at most.


Yes, but no -- I wasn't one of the Google Wave faithful so I don't know the ins and outs of it, but I what I wanted was the ability to extend SMTP all together to include the possibility of negotiating XMPP -- almost like what's happened with HTTP 1.1/2/3 evolving to support new connection types and content types/etc.


It was sometimes possible to hold realtime conversations over email, before the amount of required spam processing became too high. There's no protocol reason why you can't do that between servers, although getting it into a client requires either IMAP push or something more modern.

> make an account

You are required to make an account somewhere so your account can be banned if you spam, basically.


Apache James is a great smtp and imap server that has a lot of really great features, like distributed storage in cassandra and some support for jmap.


This is why I keep coming back to HN -- thanks for recommending Apache James.

I thought I searched far and wide for other mail servers, yet only came up with postfix, iredmail and cyrus, I clearly didn't look hard enough


Honestly I'm surprised no one has suggested replacing SMTP with a HTTP POST request. All you would need is an email address -> HTTP URL conversion (which I believe something like WebFinger already has). Then just POST your message to that URL and you're done. ISPs can run proxies that just authenticate that requests are from users and forwards them on.


Emails are perfectly fine as they are. We also don't keep re-inventing forks, umbrellas, hammers and such.


But we do. The "re-inventions" just tends to be increasingly subtle, because the high level structure continues to make sense and the overall designs are simple to begin with, which makes it hard to come up with revolutionary new ideas in the space.

Umbrella's for example see continuous evolution in means of making better fully collapsible versions, making the deployment more automated, reducing weight, etc.

From Wikipedia, illustrating both that there's continued substantial effort in coming up with new ideas, and at the same time that it is hard to come up with something genuinely new:

> Umbrellas continue to be actively developed. In the US, so many umbrella-related patents are being filed that the U.S. Patent Office employs four full-time examiners to assess them. As of 2008, the office registered 3000 active patents on umbrella-related inventions. Nonetheless, Totes, the largest American umbrella producer, has stopped accepting unsolicited proposals. Its director of umbrella development was reported as saying that while umbrellas are so ordinary that everyone thinks about them, "it's difficult to come up with an umbrella idea that hasn’t already been done."[36]

(https://en.wikipedia.org/wiki/Umbrella)


The ones being sold at the shops I regulary buy haven't changed a bit, besides their colours.


So you know that “we” don’t reinvent umbrellas, because “you” buy the same all the time?


I don't imagine who would waste their money in re-invented umbrellas, maybe it is an hipster thing.


Just as one example, wind resistant umbrellas have an actual useful feature regular umbrellas don't have.


Given that regular umbrellas cost 3 euros on average, how much do those cost?


They are significantly more, but it's not just replacement cost avoidance that's the benefit. They are easier to handle in wind, keeping you dryer and not having to fight the umbrella.


People come up with ideas for frustrating things they have a lot of contact with; seemingly every HGV driver has "invented" a way to automate matching cab (ie tractor) plates to trailer plates.

People buy cheap umbrellas, they suck, the people imagine better umbrellas that cost more: goto START.

A 9yo of my acquaintance is really inventive; he often says "what if we had something that would ...", yes, if you'd invented and developed that 50 years ago you'd have been a multi-millionaire.

A lot of inventions arise naturally out of a creative mind being confronted with the problem.


The main failure in mail standards is a lack of explicit utf8 clean support lhs@rhs -if this got fixed (it's often called universal acceptance) a lot of things about mail as an ecology would improve.

I have view on the spam thing. The whole "your idea will not work because" meme is hugely destructive of innovation in email. It sucks energy and mindshare. It's classic old timer put down.

What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

Jmap is utf8 clean btw.

(I used to do email for a living in the eighties when life was simple and bang!chains!worked)


> What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

I propose a scheme where the sender pays e.g. 5 ct per e-mail (low enough that it does not matter for legitimate use, but high enough to make spam unprofitable). BUT with the following twist: The receiver can generate API tokens that allow free e-mail delivery.

So when I sign up for e.g. Twitter, I can give them an API token for my mail address so they can send notification mails to me for free. If they decide to start spamming me, or if they decide to sell my token to a spammer, I can just revoke the token and the spam flood stops instantly.

I have not found any flaws in this idea yet. RFC!


First thought is that international differences in value of a fixed monetary amount might break it. 5¢ is affordable in USA but too much in, I don't know, Mongolia. Lower it to 0.05¢ and you get spam from USA.

Natural solution is a pricing list, but then USA spammers could route via cheapest geographic servers.

How do you get paid? Banks will put a transaction cost making sending email cost stupid money. So, bitcoin? But the built in energy costs for processing a transaction will force a floor on the transaction pricing that's too high?

If a token leaks you'll have to fiddle about to allow a sender's emails to get through ; I guess you could automate that if your MUA had credentials to inform genuine senders of API key updates.


You could do away with the 5ct part of this, just by requiring a special header to get around the api part.

Then you funnel all that junk into a special folder (black hole) that you can either ignore or check if you are expecting a legit request for your attention.

The more I think about it, the more I love this idea. All email that shows up in your inbox is there because you have to explicitly allowed it.


I've been thinking of a similar system, actually, but instead of having each end user pay per transaction, you have each domain put up a bond (in a digital currency) which can be forfeited if the domain is found guilty of spamming.

The one controversial part of this is deciding who is permitted to redeem the bond, i.e. who is the central arbiter of what is and isn't spam? Well, fortunately we already have an entity like that, de facto, since we have the Big 4 email providers ("Gmail, Hotmail, Yahoo, and AOL", or any other similar group that all mail servers must, in practice, comply with the demands of).

In the system I'm proposing, if any of these big providers detect a domain being involved in spam (or N out of M providers, to reduce the risk of false positives), they could redeem the bond. Of course, the bond would be structured such that the only recipient was a non-profit, like the IETF, or possibly an entity like ICANN. That way there is no financial incentive to make false positive claims.

As for how this system would be introduced, that's the easy part. All pre-existing domains would be grandfathered in, so that no current email users would be negatively affected. (Indeed, there is a moral hazard that some email providers would support this system precisely because it only affects new entrants to the market). There would simply be a flag day, after which if you register a new domain, and want to send emails from it, your domain registrar would have a check box saying "Yes, I am happy to be charged an extra $10 and have put it in a bond so that I can send email to the Big 4 providers". After sending a certain amount of legitimate email, a Big 4 provider could then send a transaction which makes the locked funds return to the original issuer of the bond.

Finally, to decide how big the bond has to be, we can look at data like this:

https://krebsonsecurity.com/2018/06/bad-men-at-work-please-d...

If spammers are going to extra effort to save a few dollars on their registration costs, then their margins must be pretty tight. It also means that domain reputation systems are working, since spammers have to keep bulk buying new domains:

http://conferences.sigcomm.org/imc/2013/papers/imc247-haoA.p...


If paying to send email is your suggested solution, I can see why you don't like the meme in question. But the suggestion is rife with problems. How exactly are you going to get everyone to start paying for email, which is free today? If it's a parallel system, how are you going to get people to switch? Who are they going to pay?


If people received credit for incoming mail then the majority wouldn't get a bill as most people receive more than they send.


As an alternative but similar idea, Proof of Work was originally developed to solve the email spam problem. The idea was that you would hash the email body + timestamp + nonce producing a hash with some number of leading zeros. It would be fast for individual devices to compute and fast for email servers to verify en mass but expensive if you needed to send lots of emails.

The obvious downside is that since the system is fundamentally just cryptography it doesn't discriminate between spam and legitimate bulk mail (password resets, newsletters, mailing lists, etc.)


Most bigger services are probably going to send and receive about the same amount of mail so it will be an almost zero-sum game. Most services today have a cap on how mail you can send, and this could be made more explicit and have cases where the end user actually pay.

I don't really see the payment as a big problem. This has been discussed previously, mostly initiated by the big email services


> Who are they going to pay?

The sender pays the reader. If the reader is the one deriving the value from this communication, they would pay the sender out of band as compensation.


So what does a reliable, not centralized payment system that is viable for this volume of tiny transactions that everyone can use look like? That's IMHO the big problem with all these suggestions.


> What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

Along with the billions of spam SMS that are also supposed to be sender-pays, sadly


Is this in the US? I've had SIM cards in three European countries, never received a spam SMS.


I've experienced this primarily in two European countries, as well as the Americas (both north and south) and east Asia.


> What would (imnsho opinion) have fixed spam is sender pays.

Alternate but similar - sender holds the email until the receiver fetches: https://cr.yp.to/im2000.html

I'm not sure it actually helps that much to cut down spam but anything is worth a try at this point.


That sounds like a nice way to allow the sender to work out whether the receiver mailbox is active.


That assumes there's no facility for batching - where my server can ask a remote server "give me everything for <mydomain>" rather than asking for each individual mail.


How about a combination of sender-pays and a whitelist for everyone else? Essentially, have a web-standard for what linkedin is doing.


Paying might reduce the spam, but some spammers are perfectly fine with paying. Just look at text spam and robocallers.


Sender pays to stuff junk in my physical mailbox and I have a lot more trouble with spam there than I do in my email.


You could pay with computing power.

https://en.m.wikipedia.org/wiki/Hashcash


The standard response to proof-of-work spam deterrents is that "significant numbers of senders of legitimate email would be unable to continue their current levels of activity"[0]. While I don't think that's necessarily a bad thing, it does limit the utility of such a service.

[0]: https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf


As I see it, hashcash could be a good idea if there's a standard way for the receiver to tell the sender how much hashcash they need to include.

Email is designed around asynchronous clients, e.g. the ability to write everything offline, connect for just long enough to queue them all up in some server, then disconnect while that server passes them along. By the time the receiver (server or client) knows that they've been sent a message, the sender may be long gone.

Hashcash only works if:

- It can't be bypassed. Without a mechanism to tell senders how much to include, receivers must set their price near zero to avoid discarding legitimate messages which guessed the wrong amount. Keeping messages without any/enough hashcash would defeat the purpose of the thing.

- The sending client does the work. Since servers are always online and reachable via a known address, we could have them negotiate an amount; e.g. the receiving server checks the message for hashcash, returns an error stating that a certain amount is required, the sending server tries again with more hashcash. The problem is, getting the sending server to mine hashcash won't stop spammers: they'll just use other people's servers, like gmail, hotmail, etc.

I've written about this before, but I think that a generic protocol for negotiating hashcash would be really worthwhile. Maybe it could be made to work for email, but even if not there are plenty of synchronous protocols which could use it.

In particular, there's no reason to keep a fixed price; we can figure out a price using the same heuristics as existing spam filters: can we verify the sender, have we seen spam/ham from them before, do they appear on black/whitelists, etc. This way the pressure can be kept on spammers, whilst the majority of normal traffic can go through with little effort. Note that this fixes the mailing list problem too, e.g. if users add the list address to their whitelist, or if they send a message in order to subscribe (hence triggering the "allow this, it's a reply to our message" heuristic).

I also think this would be a nice alternative to API keys, since it would keep things more "open" for experimenting and mashups, whilst giving providers a way to avoid abuse (API keys could still be provided, as a way to significantly lower the amount of hashcash required).


It seems like a good idea until you realize spammers, like DDoSers, aren't using their own machines.

The idea was dead on arrival.


Then you might at least notice some CPU usage.


I don't see why you are getting downvoted. Proof of work is a great way to limit spam. It could be made almost unnoticeable by the end user sending a couple of messages, but expensive for spam. There are of course some show stoppers (mailing list, non-profits etc), but it is not a bad solution.


That's horrible for the environment.


Depending on the workload it will be a marginal load, and getting rid of spam would be a pretty big gain.


Filtering spam mail is too - what's your argument?


So, send ten emails and your smartphone battery is dead?


To give some context on why JMAP is relevant today, you might want to read this answer in the conversations FAQ [1]. It basically states that XMPP powered Apps are having trouble fighting against the battery saving software because XMPP is stateful. JMAP, on the other hand, is stateless.

As protocols are meant to connect different implementations and bringing more protocols for the same task quickly hurts the interoperability (before bringing some improvements in the long-term), I am skeptical to the effects those new protocols bring to the ecosystem. I am aware that JMAP was born more out of the RESTful requirements of an HTTP based App, but in general, I am wondering where this road will lead us.

[1]: https://github.com/siacs/Conversations#but-why-do-i-need-a-p...


Note that "RESTful" in JMAP's case means "everything over JSON HTTP POST", methods to be invoked are embedded in the JSON request.

As for XMPP battery consumption I didn't see major problems, Conversations.im is always <1% (I just checked and it shows 0%). On the other hand Conversations can use push to optimize battery usage.


Yes, in fact, there is absolutely no reason to restrict well-built apps like conversations from consuming battery automatically. Even without Google Push Notifications the battery consumption is quite good.

So it is pretty obvious that the vendor built battery optimization software does more bad than good.


Last time I looked at IMAP, I was a little peeved that servers didn't accept messages added to the "Outbox" folder and send them, instead requiring that clients talk SMTP and then (or not) save a copy of the same message to the "Sent" folder.

Could JMAP replace SMTP as well as IMAP?


Storing sent mails is really one of the oldest problems of the SMTP/IMAP combination.

BURL [1] is an SMTP extension that allows to submit a mail that is already stored on an IMAP server. This still requires to make a separate connection to send the mail, but at least it does not require uploading it to both SMTP and IMAP only to store it in the Sent folder.

However, this extension is rarely implemented or deployed, although the RFC itself is already twelve years old. As an example, Dovecot started to support it by acting as a SMTP proxy [2] as of version 2.3 in 2017. I am not aware of any mail clients that support BURL.

[1] https://tools.ietf.org/html/rfc4468

[2] https://wiki.dovecot.org/Submission


JMAP actually does support replacing both. My next TODO item for Cypht is integrating SMTP support with JMAP.


Just curious, did you build Cyrus yourself or there is beta server already?


I checked out the git source and built it from there. It was non-trivial to get it built and configured properly for JMAP, but I wanted the latest as the spec is still in flux.


Why do you think you need a new protocol for this? Instead what you want is a different IMAP server.


It has been a long time since I looked at IMAP. Are servers that do this now commonplace? (None I could find used to.)

Can I know from a capability exchange if the remote server will send emails after adding to the "Outbox" folder?


No, not as far as I'm aware. But it's an idea I've also had. Upload the message to drafts, and then when you're ready to send, just move it to Outbox right?


That would be ideal. Keep SMTP as a server-to-server protocol only and IMAP as server-to-client.


And update all clients everywhere so that they are compatible with the new IMAP server.


The article mentions that JMAP does calendars as well. Anyone have a quick status update on how it compares to CalDav and if it's supported by any of the major calendar clients or servers? I did a quick google and looked at the JMAP site but couldn't find any info.

Background for this is that I implemented a calendar client and a calendar server (with SabreDAV) last year and I'm wondering if I should be adding support for JMAP anytime soon.


Editor of the JMAP specs here. The JMAP calendar spec is currently just a draft, but now JMAP core and mail are (more or less) finalised and JSCalendar (https://tools.ietf.org/html/draft-ietf-calext-jscalendar-11) is in last call too, we will be looking to take the JMAP Calendar spec through the IETF process in the very near future. Since it is essentially just combing core with the JSCalendar data format, this should be reasonably straight forward.


I see, thank you so much for providing inside information. Looks like it'll be a while before I can ditch CalDav...


Current state of calendaring is horrible, it's basically not possible right now to create compatible and reliable library/application.


Calendaring truly is surprisingly difficult. Dates and times just are not defined with computers in mind.


Yep, they are extremely difficult. Timezones, group events, repeatable events. RFCs describing calendaring are very vague and miss a lot of important details. And exchange/outlook implementation adds a huge pile of incompatibility.

When I was working on calendaring project I've found lots of ways to create broken events in google calendar for example. They are completely valid according to RFC but absolutely not usable in apps.

Also there are working groups that STILL developing calendaring RFCs, they create tons of esoteric and cryptic documents every year.

P.S. When I write “RFC” I mean any well established standard.


Post is about JMAP (Fastmail's IMAP/SMTP replacement), by the author of a newish webmail client called Cypht that has just recently implemented support for JMAP. Cool! Keep going!

https://cypht.org/


Loved this bit:

> Internet: Do you think JMAP will really take off?

> Me: JMAP is an open, smart, modern, and powerful E-mail protocol, so probably not.


I've started to migrate 18 years of email history to Fastmail, after running my own mail infra for almost 2 decades.

I am not impressed. Folders moved in the webinterface show up unmoved in Thunderbird. And vice versa.

Sometimes it works, sometimes it doesn't. They're investigating the problem at the moment, but the updates they've given me don't give me much hope.

I'm looking at migrating away to another provider that just does plain old imap.


From experience, Exchange's Outlook API ("EWS") is pretty decent. It's XML-SOAP, sure, but there are libraries for it that can be readily used. The only thing they did fuck up is the three different kinds of IDs for an object (esp. confusing when accessing delegated team mailboxes/calendar events) but once you get it how it works, it's straightforward and allows you access to anything from email over calendar to address-book.

I just do not know if EWS is an "open standard" that could be replicated by a third party server or if it is only "open documentation" and licensed for Exchange only :(


I've sadly been looking for, and never found, an open source EWS <-> IMAP/(Cal/Card Dav) gateway. I'd love it if my users could get a full experience using Outlook for Mac for example. Right now I recommend them to just use Apple Mail/Calendar/Contacts or Thunderbird with the tb-sync extensions.


So you'd like for users to keep their familiar Outlook setup, but with a FOSS server backend?

Not sure if this is possible, I don't know if Outlook uses the EWS protocol, OWA or something entirely different :/


Yupp. It works well with Windows Outlook, since they can use ActiveSync. But Mac Outlook doesn't have ActiveSync (or Card/Cal dav).


It sounds like you’re describing DAVMail


Wrong way. I want to offer EWS from my linux (not exchange) mail setup. Davmail consumes EWS and offers Cal/Card dav etc.


> JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect (which is exactly NOT how IMAP is supposed to be used)

WTF? So, webmail was limited by the fact that it was running in a browser and thus had to use HTTP, which has semantics that don't really fit the needs of an email access protocol. Now, we do have stuff like websockets that would make it possible to run a protocol from the webmail client that actually fits the needs, and instead people invent a new protocol that inherently doesn't match the needs of the application?


Most mail clients are running on mobile (judging by number of devices). Long-running connections on mobile are a bad idea because of network reliability and battery concerns.


Hu? Constantly establishing new connections saves power vs. maintaining one estalished connection? Could you explain how that works?

And could you also explain how constantly making new connections makes things work more reliably over unreliable links? Like, does that allow you to transfer data when the network link is down? Does the fact that inside the TLS/TCP connection data is transferred via HTTP instead of IMAP somehow make the TCP connection work better over lossy links?

It would seem to me like the exact opposite should be the case, if it has any effect at all?


Ignoring efficiency concerns, modern mobile oses don't allow applications to keep connections open for long. If you want notifications to work, you need to use the platform notification system.

The platform argument is roughly that if they hold open one connection, it's cheaper than each app holding open their own connection. And maybe, if you're optimistic, the platform will be better at figuring out ping intervals for that connection that adapt to broken Nat devices per network (ie: on some networks, wake up once a minute to keep the connection active, and on reasonable networks wake up once every 30 minutes)


Well, yeah, if you are very optimistic and generous. In reality, this is yet another way of the manufacturers maintaining power over their customers' property. After all, there is zero reason why any of those things would require them as a middle-man, they could just invent a protocol for that and/or provide a library and a system service on the device that allow applications to maintain a background connection without the need to involve them in any way on the network/server side. The only useful involvement they could have might be providing endpoints for measuring and exchanging NAT timeout information for providers, maybe.

But in any case, none of that is an argument for building a protocol that forces the problems that result from that on all platforms that don't have such restrictions, at best that is an argument in favor of workarounds to make things work as well as possible on such platforms.

Also, I would think that the long-term solution to this problem should be mobile link layer protocols that allow the mobile station to receive incoming packets with little delay while still conserving power. Stuff like high-powered, high SNR alert channels that allow the mobile station to shut down everything apart from a simple, low power, receiver, that only powers on for a few microseconds every few hundred milliseconds to listen for a wakeup signal from the network, so that incoming packets can be delivered with a latency of a few hundred milliseconds at any time. And then you are stuck with a stupid poll-based protocol that's adapted to the limitations of some ancient technology as a replacement for an even more ancient protocol that didn't have that limitation.


Webmail was limited by HTTP, but not because HTTP is stateless; rather, it's because no mail server speaks HTTP, which means every webmail has to implement it's own IMAP proxy, and IMAP as a protocol is very, very bad if you don't want to sync emails to your local machine and thus keep your own copy of the whole mailbox state.

On the other hand, JMAP does allow you to get push notifications on changes, outside of the normal HTTP API.


> Webmail was limited by HTTP, but not because HTTP is stateless; rather, it's because no mail server speaks HTTP, which means every webmail has to implement it's own IMAP proxy

That doesn't explain why it is limited, only why it isn't interoperable. A traditional IMAP client also didn't speak TLS. That wasn't a reason to invent a replacement for IMAP, because you simply stack IMAP on top of TLS, and noone complains that "every TLS IMAP server has to implement its own TLS proxy", let alone gets the idea that you should invent a replacement for IMAP in order to be able to use it over TLS.

> and IMAP as a protocol is very, very bad if you don't want to sync emails to your local machine and thus keep your own copy of the whole mailbox state.

Because? And mind you, we are looking for a reason that would justify (a) inventing a whole new protocol instead of adding a few small extensions and (b) specifically inventing a pull-based protocol that uses HTTP as the basis.

> On the other hand, JMAP does allow you to get push notifications on changes, outside of the normal HTTP API.

In other words: Because JMAP doesn't fit the needs of the application, because it uses a pull-based protocol, and in contrast to IMAP which has push built-in already (well, it's an extension to the original protocol, but one that is widely supported and a relatively simple change to the protocol), they support tacking on a workaround for the resulting problems? And that is supposed to be an argument for the protocol, or what?


You obviously disagree, but JMAP made exactly the right choice: A simple, pull based protocol, with a side-channel for notifications. It's not clear why anyone would want a complex websockets-based monstrosity when the simple approach solves the problem perfectly.

I am not at all convinced that JMAP will succeed, I think it's along shot, but it's failure will not be because of the choice of HTTP.


What exactly is the point of simply ignoring what I wrote and making unsubstantiated assertions that contradict what I wrote?

First, it's based on HTTP, so it's obviously not simple. Then, it obviously doesn't solve the problem perfectly if it cannot do the job without an additional side channel (which still doesn't solve the problem perfectly as the side channel obviously is slower and reduces reliability). And what is possibly the monstrosity with websockets, if not the fact that it is kinda-sorta using HTTP for something it wasn't built for ... which you think is best avoided by using HTTP for something it wasn't built for?

And no, I certainly don't want to use websockets, just as I don't want to use any of the other monstrosities that make up "email in a browser". But the great thing about sensible protocols for the purpose, which could work over websockets, is that you can simply drop all that crap and run them over TCP (with TLS in between, preferably).


> Constantly establishing new connections saves power vs. maintaining one estalished connection? Could you explain how that works?

If you maintain a connection you have a leave the modem powered. If you connect every so often (1 minute, 10 minutes, whatever) the modem can be powered down in between. The modem and the screen are the top two users of power in a phone so this is a big win.


Sorry, but that makes no sense. Keeping a TCP connection in established state does not require any modem to be powered. I even suspend my laptop with established SSH connections over night, to disk, so it's completely powered off, I could even disconnect the power supply and take out the battery for the night. When I power it back on, the SSH sessions are still working perfectly fine, after hours of being powered down, and that is completely unsurprising, given how TCP works.

The only thing that requires leaving the modem powered on is if you want to be able to receive something. But tearing down the connection obviously doesn't help with that. When the modem is powered down, you won't be able to receive anything more by not having an established TCP connection.

So, no, that does not explain how doing more work saves power.


That does not work over arbitrary connections, especially not mobile ones: in many cases, long-running connections are terminated (either because a moving phone moved in the network and received a new address, or because a middlebox threw away state about a connection it hasn't seen traffic for in x minutes). If you shut the modem down, you also can not receive messages, so you can only use polling patterns anyways (at which point the benefit of e.g. Websockets is largely lost).

So in practice, you have to regularly send traffic down the connection, fully powering up the modem, and ideally that shouldn't happen for each app individually. Now one could imagine a system where the OS coordinates this and signals to apps that the system is going to transmit now, and that they should trigger their own requests, bundling them in the process, but that's not how the platforms work. Instead, the choosen method is to have central push services, which the phone OS polls and then distributes the messages to the apps.


> Now one could imagine a system where the OS coordinates this and signals to apps that the system is going to transmit now, and that they should trigger their own requests, bundling them in the process

FWIW this would happen quite naturally if the system simply used flexible timers for these sorts of wakeups. It's an accepted, even trivial solution to the "need to power something up on an infrequent and perhaps unpredictable basis, and have it be used seamlessly by multiple apps" use case. Centralized services are nice but they shouldn't be the only way for a system to get notified about stuff.


> That does not work over arbitrary connections, especially not mobile ones: in many cases, long-running connections are terminated (either because a moving phone moved in the network and received a new address, or because a middlebox threw away state about a connection it hasn't seen traffic for in x minutes). If you shut the modem down, you also can not receive messages, so you can only use polling patterns anyways (at which point the benefit of e.g. Websockets is largely lost).

That's only true in so far as it is besides the point. Yes, suspending your phone for hours on end and expecting established TCP connections through some telco's IP connectivity to still work obviously won't work. But that wasn't my point. My point was that interrupting connectivity does not interrupt TCP connections, or other kinds of long-running connections. And that tearing down the connection does not in any way improve the situation with regards to not being able to receive messages.

The mere fact that polling doesn't make things worse in a particular use case is not an argument for designing a poll-based protocol. But also, saying that you can only use polling patterns anyway is a massive oversimplification. Even if you implement a poll-based "background delivery" mechanism, a push-based protocol still is advantageous for when the application is being actively used in the foreground. Mind you, you generally can still poll with a push-based protocol, but that does not really work the other way around. Even if you poll for new messages in the background, you have a better user experience when server-side state changes during active use of an application are reflected immediately in the user interface, so you should still have a push-based protocol, and only switch to polling when in the background.

Also, part of your argument only applies to TCP, which unfortunately uses the IP addresses as part of the connection identity, but that is not an argument against long-running connections per se, as you obviously can build connection protocols that don't do that, see QUIC for a real-world example. So, if that is an aspect you want to solve, you shouldn't invent a new poll-based protocol on top of a "transport" that doesn't fit the needs of the application and that also doesn't even solve the problem (when the IP address changes during an HTTP request, that connection will still fail, possibly using a long-ish timeout, and in any case necessitating a complete retransmit of anything that had already been transferred), but instead you should maybe write a specification for IMAP over QUIC, so you can actually seamlessly continue a data transfer over address changes and NAT remappings.

> but that's not how the platforms work. Instead, the choosen method is to have central push services, which the phone OS polls and then distributes the messages to the apps.

Well, yeah, but that's simply the manufacturers forcing a technically inferior solution on their users in order to gain power over their property. While designing a workaround for this situation certainly is a good idea, making this idiocy the basis for a protocol design, so that even software on sane platforms can not do better does not exactly seem like a bright idea.


I was really hoping for a GraphQL API standard for mail before reading this. But this sounds good too.


Understand that JMAP is an object synchronisation protocol. It’s most interested in the problems that are out of scope for GraphQL.

You could have a variant of JMAP with GraphQL syntax and semantics, but there would be a fair bit of mismatch, for their purposes and focuses are quite different: JMAP is concerned with object identity and synchronisation, for what you might call thick clients (and JMAP Mail needs to be broadly IMAP compatible), while GraphQL is UI-focused, for what might reasonably be considered to be thin clients (not that they are logicless, but that they are very much focused on offloading burden to the server). I see no compelling case for a JMAP-like GraphQL thing.

It could still be an interesting task to develop a object synchronisation protocol like the JMAP core on top of GraphQL.


Reading through the JMAP website [0], I'm actually wondering whether whoever's behind JMAP has looked at GraphQL. There seem to be a lot of similarities (especially the "why not REST" section).

[0] https://jmap.io/index.html


Work on JMAP likely started before GraphQL was publicly released.


I think Fastmail are the primary ones behind JMAP: https://fastmail.blog/2014/12/23/jmap-a-better-way-to-email/


Yep. Also I think they are using it for a subset of customers. While JMAP support in Cypht is still new, so far it's working just like IMAP, but my testing is limited since I just wrote the integration :)


Actually, Outlook GraphQL reusing the idea of batch JMAP calls. https://docs.microsoft.com/en-us/previous-versions/office/of...


Very interesting:

> JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect

This would allow implementing JMAP in serverless architecture, e.g. self hosting on AWS lambda. That should be very cheap and significantly lower the barrier to self hosting (not needing to manage a box, just providing aws credentials).

Think of what it takes to truly decentralise email. There are technical hurdles, but also some fundamental ones:

- other providers mark you as spam because you’re unknown

- you need an always on server to actually get your incoming mail

This would at least solve the second problem. If we can develop a product like mailinabox (Which has its flaws but it’s the right idea), but instead of asking for a fresh vm, it just asks for aws credentials, that could be pretty solid.

Hopefully one day we can give the people back their control over their means of communication. This seems like a step in the right direction!


I'm not sure I understand: JMAP is designed for communication between the client and the mail server. But you still would need a mail server, always on, to receive the incoming mail from other servers, right ?

Did I miss something about JMAP?


Nope, you are exactly correct. JMAP is just a much more modern and efficient way to connect clients and servers to access E-mail.


Why?

Mail is decentralized already. Part from relying on DNS.

And you seem very eager to put everything in the amazon basket.


If everyone hosts their email with Amazon it's gonna be quite centralised.


The first problem is actually the more serious one, sadly.


So, you say that it doesn't maintain a persistent connection, but that poses a problem - it needs a way to do push notifications still I assume? Those need an open socket somewhere - does JMAP allow for it or does it make you rely on a 3rd party?


If you are mention Ios/Android, yes you need the proxy to redirect push calls from JMAP server to the push services of apple/google.


FWIW this is completely reasonable. Push in XMPP works the same way. They way FCM (Google) and APNS (Apple) work is that only the app vendor with proper credentials can trigger push notification for an app. Therefor it has to be proxied through the app developers server anyway. And of course it cleanly separates the MDA from the various, proprietary push solutions.


That sucks, I'd like to involve as few parties as possible in my emails. Right now I just have a direct IMAP connection that it can push over even on mobile.

What about for desktop purposes? What would a proper client like Thunderbird or Outlook do?


IMAP is just a way for attackers to perform credential spray's against large providers these days.


I recently thought about email being based on such an old standard and then I wondered if maybe it didn't matter anymore, because most of the web is moving to gmail, and my guess is (and somebody please correct me) that when I send an email from gmail to gmail, that no actual SMTP is involved? I'm assuming that they have some internal protocol that is just adapter'd to "Email" at the very end?


> most of the web is moving to gmail

This isn't true; there's a LOT of industry on Exchange of some sort, and plenty of older institutions running their own email. Especially in IETF land.

And it's a huge risk increase if everyone moves everything to gmail.


It fakes it, at the very least - you can do a “show original message” and see all the RFC.822 and mine headers.

Microsoft Exchange used to not bother with actually SMTPing when talking to another exchange server (or itself), I don’t know what it does these days.


I hear of a lot more people moving away from Gmail than moving to Gmail.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: