
JMAP: A modern, open email protocol - okket
https://www.ietf.org/blog/jmap/
======
ocdtrekkie
One of the things I particularly recall reading is FastMail devs explaining
that the standardization process led to changes and improvements. (Even their
about page says that "JMAP was built by the community, and continues to
_improve via_ the IETF standardization process.")

Really thrilled the FastMail team is working on this, and doing it in the way
that changes to mail should be handled: Going through proper Internet
standards bodies. It's in stark contrast to AMP4Email, which is being
implemented by Google wholly without respect to Internet standards.

~~~
contingencies
_Really thrilled the FastMail team is working on this_

Sorry to be 'that guy'. I like Fastmail but maybe they could fix some of the
irritating bugs we keep asking for resolutions on before embarking on fun
protocol invention missions. Two killers:

(1) When forwarding an email there is no link between the forwarded email and
the original thread (TOTAL TIME WASTER), when reported they "don't consider it
a bug [because of SMTP ID handling in existing daemons]". Missing the point
here so hard guys!

(2) When creating a new filter it doesn't have a checkbox to include matching
emails. This means you have to manually go replicate the search again, for
consistency, to get things where they should be.

Frustrated at lack of response from reporting these as a commercial user.
Inventing new protocols should not be higher priority than fundamental
annoyances that are wasting customer time NOW.

~~~
Aeolun
Did you consider that they might only be wasting _your_ time? I haven’t ever
felt any frustration at any of the things you named.

~~~
contingencies
Both are standard features on GMail, which beats your sample size of 1.

~~~
kljdsjiodjaof
And I didn't care about them there either.

But you know, you could send them the feature requests - they actually
implemented something I asked for on mobile, so at least one team is
listening.

------
satyanash
If this takes off, it would remove one of the last few holdouts of non-HTTP
application specific protocols. That just leaves us with SMTP really. While
using HTTP is not necessarily a bad thing, it's certainly an interesting
indication of how application developers prefer convention based protocol
definitions (JSON over HTTP) instead of having a strictly defined protocol
over a duplex stream (IMAP over TCP).

~~~
jolmg
Why is this happening? Forgetting ubiquity of tooling, is there something
technical about HTTP that makes it so attractive?

I get JSON. People don't want to write text-parsers. But what's attractive
about this abuse of HTTP? This isn't hypertext.

Is it so complicated to work with a duplex-stream? Could a library alleviate
the complications?

Could it be that developers are more familiar with HTTP than with TCP and so
TCP scares them even though it's simpler to use?

~~~
zeroxfe
> This isn't hypertext.

That ship has long sailed. HTTP is no longer just hypertext -- it's a
ubiquitous and well-understood transport for RPCs, with lots of tooling and
infrastructure support across the Internet.

I think at its core, HTTP abstracts away some of what you would want in a
request-response protocol, but not really all of it. But _simply because of
the ubiquity of tooling_, you get a whole lot of other stuff for free (load
balancing, DDoS detection, in-browser debugging, etc.)

> Is it so complicated to work with a duplex-stream? Could a library alleviate
> the complications?

Yes, TCP streams are very complicated to work with (in my experience) -- there
are way too many failure modes and corner cases to consider. IMO, your
language's go-to HTTP library is that library that alleviates the
complications.

All this said, I personally think gRPC is the right approach for RPCs, and I
hope to see broader adoption of it over the coming years.

~~~
jolmg
> Yes, TCP streams are very complicated to work with (in my experience) --
> there are way too many failure modes and corner cases to consider.

Isn't it the same thing, but simpler and more flexible? You write a message,
and then you read a reply, rinse and repeat. Could you expand on such things
that are complicated with TCP that HTTP helps with?

~~~
zeroxfe
When you're working with sockets there's lots to think about that you can
simply ignore with an HTTP library -- connection setup and teardown,
buffering, serialization, byte ordering etc. But more importantly, you'll have
to build your own status/error handling, cache control, content encoding, TLS
etc. (Simply, doing TLS correctly itself is not easy.)

To be clear, I'm not saying that there's no place for socket programming --
there definitely is. I've spent a good part of my life doing obscure network
programming on all kinds of platforms, so I try not to use TCP/UDP unless I
have a really good reason. (It's kinda the same reason I won't use C or C++ --
too much to worry about for most general purposes.)

~~~
jolmg
Thanks for the reply. This gives me a better understanding of the issue here.

> connection setup and teardown

If there's no benefit in avoiding setting up and tearing down a connection
with each request like one typically does with HTTP, then I can see how it
might be an annoyance compared to making a single function call from a library
that makes the request and returns the response.

> buffering

A parser should deal with that. It could be JSON parser.

> serialization

That's still a problem with HTTP. People typically use JSON for this and it
doesn't depend on HTTP.

> byte ordering

parser/JSON's job.

> your own status/error handling, cache control, content encoding

That can be beneficial or it can be problematic. If you need such things, then
yeah, it's cool to not have to think about making your own. However, if you
don't or if your needs don't align with what HTTP provides, that can be
problematic. For example, from a cursory search I see the following in the
JMAP spec:

> Implementors must take care to avoid inappropriate caching of the session
> object at the HTTP layer. Since the client should only refetch when it
> detects there is a change (via the sessionState property of an API
> response), it is RECOMMENDED to disable HTTP caching altogether, for example
> by setting `Cache-Control: no-cache, no-store, must-revalidate` on the
> response.

So, HTTP provides caching, but not the caching mechanism that's needed, so
it's recommended to explicitly disable it completely.

With respect to status and error handling, it seems JMAP still had to define
their own errors to be described in JSON[1], so not a perfect fit there
either.

On the content encoding, I'm not sure if JMAP would use something other than
application/json, so it seems like something that wouldn't need to be
specified if one didn't use HTTP.

I know the discussion is not really about JMAP. I'm just using JMAP as an
example on the above.

> TLS etc. (Simply, doing TLS correctly itself is not easy.)

I would hope that there's a library equivalent of the shell's:

    
    
      openssl s_client -connect $host:$port
    

I don't know why a library would have to be much more complicated. Maybe I'm
being naive, but I think that if one wouldn't specify special TLS parameters
to an HTTP library, then there's no reason why one would need to if they skip
the HTTP library and go straight to using a TLS library. IOW, I don't see why
an HTTP library would be more helpful on setting up TLS than using a TLS
library directly.

[1] [https://jmap.io/spec-core.html#initial-jmap-error-codes-
regi...](https://jmap.io/spec-core.html#initial-jmap-error-codes-registry)

------
westurner
What are the optimizations in JMAP that make it faster than, say, Solid? Solid
is built on a bunch of W3C Web, Security, and Linked Data Standards; LDP:
Linked Data Protocol, JSON-LD: JSON Linked Data, WebID-TLS, REST, WebSockets,
LDN: Linked Data Notifications. [1][2] Different worlds, I suppose.

There's no reason you couldn't represent RFC5322 data with RDF as JSONLD.
There's now a way to do streaming JSON-LD.

LDP does paging and querying.

Solid supports pubsub with WebSockets and LDN. It may or may not (yet?) be as
efficient for synchronization as JMAP, but it's definitely designed for all
types of objects with linked data web standards; and client APIs can just
parse JSON-LD.

[1] [https://github.com/solid/information#solid-
specifications](https://github.com/solid/information#solid-specifications)

[2] [https://github.com/solid/solid-
spec/issues/123](https://github.com/solid/solid-spec/issues/123) "WebSockets
and HTTP/2" SSE (Server-Side Events)

[https://jmap.io/](https://jmap.io/)

JMAP: JSON Meta Application Protocol
[https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol](https://en.wikipedia.org/wiki/JSON_Meta_Application_Protocol)

Is there a OpenAPI Specification for JMAP? There are a bunch of tools for
Swagger / OpenAPIs: DRY interactive API docs, server implementations, code
generators: [https://swagger.io/tools/open-
source/](https://swagger.io/tools/open-source/)
[https://openapi.tools/](https://openapi.tools/)

Does JMAP support labels; such that I don't need to download a message and an
attachment and mark it as read twice like labels over IMAP?

How does this integrate with webauthn; is that a different layer?

(edit) Other email things: openpgpjs; Web Key Directory /.well-
known/openpgpkey/*; if there's no webserver on the MX domain, you can use the
ACME DNS challenge to get free 3-month certs from LetsEncrypt.

[https://wiki.gnupg.org/WKD](https://wiki.gnupg.org/WKD)

~~~
dragonne
> There's no reason you couldn't represent RFC5322 data with RDF as JSONLD.
> There's now a way to do streaming JSON-LD.

Is there a reason you'd want to? I clicked all your links but I still have no
idea what Solid is.

~~~
WhatIsDukkha
Here's another link to click -

[https://en.wikipedia.org/wiki/Solid_(web_decentralization_pr...](https://en.wikipedia.org/wiki/Solid_\(web_decentralization_project\))

------
jwr
I clicked on this with a lot of hope — alas, this only seems to address the
"mailbox access" part of E-mail. I think that part actually works fairly well
today (not optimally, but well enough in practice). The part that really needs
fixing is SMTP.

~~~
jcranmer
IMAP is a really broken protocol; I'd characterize it as a database
synchronization protocol that wasn't designed to be one. You need to rely on
certain extensions to have a hope of being bug-free [1], and even those can be
problematic (UIDVALIDITY changed, gotta clear your entire inbox!).

[1] The IMAP protocol used to rely heavily on message sequence numbers, where
each message in the folder was numbered sequentially from 1-N, and deleting
messages caused all subsequent messages to have their number decremented by
one. But you can have multiple connections active on the same mailbox, and you
can't always immediately notify the other connections of when things get
deleted, so every connection can have a slightly different mapping. So
everyone uses UIDs now, which are guaranteed to be unique, until the server
feels like deleting them all and restarting from scratch (i.e., updating
UIDVALIDITY), so every message has a different UID now.

~~~
sdegutis
Where UID is used now sounds like a good use case for UUIDs (or GUIDs?) that
don't ever change.

~~~
jcranmer
I don't specifically know the history of IMAP, particularly IMAP UID, but the
semantics of UID to me implies that it was originally specified to permit its
implementation as the file offsets into the mbox files that backed the store,
and I believe some IMAP servers implemented it that way.

Some IMAP server implementations have a custom attribute they use to indicate
a globally-unique message ID.

------
cvwright
This is very exciting! It's been a long time coming.

Should we take this IETF blog post as an indication that the standardization
work is almost done?

~~~
tialaramex
The core work is done, yes, draft-ietf-jmap-core-17 and draft-ietf-jmap-
mail-16 [edit: fixed draft name] are in the RFC editor's queue, how long it
takes to get through that queue can depend on a lot of variables. At the end
it goes to "AUTH48" which is notionally 48 hours long but a few weeks is more
likely in practice, and then it gets cut a number and becomes an RFC.

The Working Group still has open items, including calendaring (because people
never can build a mail system that doesn't involve calendaring for some
reason), handling "read receipt" type features and because this is Current
Year a mechanism to involve WebSockets. Either those will eventually also be
polished into standards, or they'll be abandoned at some point and
undelivered.

------
voidcommonstock
There is so much NIH going on in this protocol, it's as if the authors we're
completely unaware of modern HTTP API design.

They're not just doing IMAP over http, they invented their own error bodies,
schema's , negotiation and RPC.

~~~
greggyb
I think their goal is to have transport layer be a pluggable subsystem,
divorced from their request-response protocol.

They even essentially say that HTTP is not necessary to their protocol, so I
think it's pretty apparent that HTTP is a utilitarian choice, but they're not
trying to build email-over-http.

~~~
specialist
_”pluggable subsystem”_

SOAP had same rationalization.

------
upofadown
Article uses the word "modern" 5 times, including in the title. Sort of a pet
peeve of mine for this sort of thing. New does not imply better.

Article also mentions that JSON is used...

~~~
afiori
What drawbacks do you see in JSON here? For machine to machine communication
it is a pretty good serialization format I would say (It is not great for many
other things, but this sound like its strong point)

~~~
upofadown
It seemed to me that the article was only using the fact of JSON use as more
evidence of modernness. It implied that JSON didn't map that well into their
problem space but they overcame.

~~~
afiori
In a sense they are right. Modernness in the sense that it is a well
understood format and not a DSL with non-trivial semantics and parsing.

------
Nasreddin_Hodja
[https://jmap.io/#push-mechanism](https://jmap.io/#push-mechanism) JMAP going
to use rfc8030 push for mobile and EventSource (kind of http long polling) for
desktop clients, as I understood. I don't like that idea, it's even worse than
IMAP4 IDLE. Why not just WebSockets for mobile and desktop clients?

------
flukus
Looking at the request it contains properties like collapseThreads and sort,
moving this sort of functionality server side is a regression. This is an API
for hosted services, it's disingenuous to compare it to imap.

~~~
crest
Do you see anything that should (or could) be done via IMAP but not via JMAP?
Adding support for higher level operations to the protocol can be worth the
complexity if it removes network roundtrips. Not everyone is lucky enough to
have a reliable, fast, low latency link to his IMAP server.

~~~
flukus
> Not everyone is lucky enough to have a reliable, fast, low latency link to
> his IMAP server.

This is a protocol from the 80's, I'm sure it doesn't require anything we'd
consider a fast connection today, latency may have changed though, mail
servers used to be geographically closer.

> Adding support for higher level operations to the protocol can be worth the
> complexity if it removes network roundtrips.

It also benefits SaaS providers by having the complexity on their end and
potentially limits apps. In my example of "collapseThreads" why does this need
to be an option at all instead of something like "parentId" and leaving it up
to the client?

More features also means more ways to abuse them, imagine google saying "yes
we support JMAP in gmail" then ignoring certain features.

~~~
afiori
> This is a protocol from the 80's, I'm sure it doesn't require anything we'd
> consider a fast connection today,

But it does require long lasting stable connections, which is what you want to
avoid on a mobile that tries to connect for only a few second at a time and
the go back to low-power mode.

------
sjellis
Getting a FastMail subscription over a decade ago really has been one of the
best tech decisions that I ever made. They are a great mail provider, and I am
very happy that my money helps fund work like JMAP.

------
xvilka
Hopefully Microsoft Exchange protocol will become a thing of a past now. It is
always the biggest problem of Linux desktop integration in Windows-only
corporate ecosystem.

------
RZTHsY5jlFA7
Almost all email clients are no longer actively developed. Hardly anyone is
going to implement it.

------
popzq
Unpopular prediction: Gmail will never implement JMAP, so it's already dead in
the water.

~~~
ocdtrekkie
Why wouldn't Google implement JMAP? Interestingly enough JMAP adds support for
Gmail-specific behaviors like labels, which IMAP lacks. This would allow other
apps and services to better integrate with Gmail's quirky behavior, improving
the experience for Gmail users.

~~~
lucideer
IMAP doesn't lack Gmail's "labels", Gmail just never bothered implementing
their labels as IMAP labels, leading to every IMAP client requiring horrible
hacks to work around Gmail's inane implementation.

Google are unlikely to implement JMAP for the same reason they've:

\- ignored pleas to implement IMAP properly going back to its inception

\- have IMAP disabled by default for new users, and for most corporate users
(requiring a GSuite admin to explicitly enable IMAP if employees want to
access their corporate email with a client).

Google benefit massively from people not using IMAP clients and being forced
into their product UIs.

That said, I wouldn't call it dead in the water for this reason. Google's
tactics here have reduced the overall usage of IMAP clients, but they also
serve as strong motivation for that small niche of users to move away from
Gmail.

------
butz
Does it solve SPAM problem?

~~~
jolmg
I haven't read JMAP, but I don't think that's the problem it's tackling.

Also, I think SPAM is as solved as it can be with blacklists, heuristics-based
filtering software like spamassasin, and personal filtering with sieve
scripts. What else would anyone suggest?

~~~
dcbadacd
Spam could be half-fixed with forcing DKIM, SPF, DMARC and TLS. The very least
there would be no impersonation of other domain names (filtering the rest into
a spam box will become much easier after that) and insecure transportation
then.

~~~
ahje
How would sender authentication solve the problem? A large amount of spam is
sent from throwaway domains with SPF and DKIM in place, and another large part
of it is sent through hijacked web sites and email accounts.

~~~
dcbadacd
As I have said, it would very least avoid spam looking real from real domains,
which is half of the issue. The throwaway part could be fixed by making spam
illegal and actually cracking down on it.

