
The JSON Meta Application Protocol (JMAP) - kuba-orlik
https://tools.ietf.org/html/rfc8620
======
chrismorgan
For context: RFC 8620 is the JMAP _core_ , just published. It defines an RPC
framework, an object synchronisation protocol, and miscellaneous other things
like push channels. It has no data model associated with it, beyond handling
blobs.

The JMAP _mail_ spec, which is what most people will be more interested in, is
awaiting final sign-off before publication as RFC 8621.

We’re using JMAP in production at Fastmail on our Fastmail and Topicbox
products, right through the stack. Both of their web interfaces speak JMAP to
the server. Unfortunately at this time it’s not ready for public use:
definition of authentication technique is one thing that is deliberately left
out of JMAP, and we’re currently using our own system which has no public
documentation (and we don’t expose /.well-known/jmap yet either). The intrepid
user may be able to figure it out, but we won’t be providing any support for
it yet. We use the JMAP mail spec, the draft calendar and contacts specs, and
a handful of our own specs for our own data types, also not publicly
documented at this time.

JMAP is much more complex than the standard sort of simple REST API, but there
is cause for all aspects of its complexity; the most notable complexity in it
is its record state management, part of it being an object synchronisation
protocol; an advantage of that is that robust offline support is feasible,
even fairly straightforward. I’m looking forward to working on offline support
in the Fastmail web app.

I personally think the increased complexity is well worth-while, and I have
several products I’ve been sketching out and working on personally (completely
unrelated to the world of email or to my employer Fastmail), and all the ones
with any kind of client-server API I intend to build atop JMAP, because
they’ll benefit heavily from it with respect to features like real-time
interface updates and offline support (for web, desktop _and command line_
programs).

~~~
lloeki
I've hacked IMAP long enough to want it thoroughly dead, so I've been
following JMAP from afar and waiting for Fastmail to publish this for a long
time. Props to the team, and I hope the (publicly) missing parts will come
soon enough.

~~~
chrismorgan
Note that IETF working groups publish drafts all along the way, so there’s
[https://datatracker.ietf.org/doc/draft-ietf-jmap-
mail/](https://datatracker.ietf.org/doc/draft-ietf-jmap-mail/) for the current
text of the mail spec, which is probably what will be published.

------
jorangreef
IMAP's killer design flaw was that it imposed a strict UID consistency
requirement between replica servers if client sync was to work correctly, i.e.
you needed a master-slave setup or Paxos, you couldn't use a true multi-master
setup or CRDTs with eventual consistency because of the possibility that
clients might miss UIDs.

There have been attempts to solve this with even-odd UIDs etc. but those broke
down on further analysis. There have also been papers claiming to implement
multi-master IMAP servers over CRDTs but these never addressed the strict UID
consistency requirement and never tested with actual client implementations
for sync correctness.

How does JMAP address the distributed multi-master server sync problem,
assuming eventual consistency?

~~~
scottlamb
> IMAP's killer design flaw was that it imposed a strict UID consistency
> requirement between replica servers if client sync was to work correctly,
> i.e. you needed a master-slave setup or Paxos, you couldn't use a true
> multi-master setup or CRDTs with eventual consistency because of the
> possibility that clients might miss UIDs.

+1, and it's been a while since I've dealt with this, so my memory's fuzzy,
but I think it's even worse than that in a couple ways:

* The failure mode here is not just that the client misses UIDs but that a given mailbox/uidvalidity/uid means something different to the client than to this replica. Then when the client says to delete one message, they delete a different one. This would be a lot less likely if the server could use a uuid for a message (so collisions are unlikely) instead of following a sequence and/or if it the identifier didn't need to change whenever it moves to another folder. (My memory is even fuzzier here, but iirc in gmail's case, its emulation of the folder model also means a message can have multiple uids at a time, one per label. That mismatch between the folder setup clients expect and the label model is another problem with IMAP. Strictly speaking, I think you can do it all with one folder and an IMAP flag for each label; it's just that the clients don't expect it to work this way.)

* Even with a synchronous replication design, there's the possibility that due to a bug or bad machine the server replicas won't match entirely, or just the client state doesn't match. And if you discover and fix such a problem on the server side, your only option for addressing the bad client state is to bump uidvalidity, which is super annoying to the client. It'd be nicer if you could do something that kicks off some softer sync that doesn't use as much bandwidth, lets the client be more usable while it's happening, and/or lets the server know what mismatches were actually found for diagnostic purposes.

I haven't looked into JMAP before, but I hope it materially improves this
situation...

~~~
nmjenkins
Author of JMAP here. * JMAP uses opaque ids rather than sequential ids, which
makes a lot of this easier (the servers can ensure they have their own id
namespace to avoid collisions). * The resync mechanism (see the "/changes"
method) just indicates something may have changed and then you fetch those
records. It's definitely possible to make your master-master replication work
so that /changes works regardless of which master you were previously talking
to, at the expense of some loss of efficiency (you may end up refetching some
data you actually already had).

------
andrewvc
The introduction, for those not wanting to click too deeply into the link:

    
    
       The JSON Meta Application Protocol (JMAP) is used for synchronising
       data, such as mail, calendars, or contacts, between a client and a
       server.  It is optimised for mobile and web environments and aims to
       provide a consistent interface to different data types.
    
       This specification is for the generic mechanism of data
       synchronisation.  Further specifications define the data models for
       different data types that may be synchronised via JMAP.
    
       JMAP is designed to make efficient use of limited network resources.
       Multiple API calls may be batched in a single request to the server,
       reducing round trips and improving battery life on mobile devices.
       Push connections remove the need for polling, and an efficient delta
       update mechanism ensures a minimum amount of data is transferred.
    
       JMAP is designed to be horizontally scalable to a very large number
       of users.  This is facilitated by separate endpoints for users after
       login, the separation of binary and structured data, and a data model
       for sharing that does not allow data dependencies between accounts.

------
rhabarba
JSON is an incredibly inefficient format for shareable data: it is annoying to
write, unsafe to parse and it even comes with a lot of overhead (colons,
quotes, brackets and the like).

I'd prefer s-expressions.

~~~
chrismorgan
I’m a developer at Fastmail, and I’m going to bite.

 _Personally_ , I prefer binary formats where feasible, because I like
pursuing efficiency to extremes, and write all my own code in Rust.

Professionally, I happen to write mostly JavaScript, and no Rust at all, but
we do still focus on performance at Fastmail. (It’s in the name!)

If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re
both “inefficient”, both representing generic data structures in regular text.
They use slightly different punctuation, but that’s about it. The differences
are negligible.

But when it comes to support, JSON is well-supported in all mainstream
languages, while s-expressions have no central specification on precise
syntax, and you’ll need to do your own thing. One of the deliberate design
advantages of JMAP over IMAP is _doing away_ with custom parsers, and using
the standard tools people are familiar with, HTTPS and JSON.

When talking efficiency, you might then ask why not to use Cap’n Proto,
protobufs or one of those sorts of things. Due to the need for extensibility
(e.g. in Fastmail we add a few custom fields to JMAP core and mail data
structures), rigid structures are unsuitable; you do need arbitrary key-value
arrangement, despite it being larger over the wire.

All in all, JSON is a robust choice for just about any API.

Fun fact: at Fastmail we compress both request and response, even using a
custom dictionary primed on real JMAP stuff, to improve the compression ratio.
I understand standard DEFLATE was about equivalent in size to CBOR (a binary
representation of JSON), and the 2KB dictionary brought it down even further,
can’t remember how much it was off the top of my head. Neil Jenkins gave a
talk about this and related matters last month,
[https://www.webdirections.org/code/speakers/neil-
jenkins.php](https://www.webdirections.org/code/speakers/neil-jenkins.php), no
public video sadly. I’ve thought about writing a blog post about it all. An
unfortunate side-effect of this compression is that you can’t inspect traffic
in the network panel of the dev tools; you can add ?compress=0 to disable it,
if you’re an interested Fastmail user.

~~~
zeveb
> s-expressions have no central specification on precise syntax

There is:
[https://people.csail.mit.edu/rivest/Sexp.txt](https://people.csail.mit.edu/rivest/Sexp.txt)

It's imperfect, but it does get the job done. Among its advantages is that
it's _both_ pure text and binary: there's a canonical representation which is
binary, and an advanced representation which is textual, and they map 1:1 to
one another.

Examples:

    
    
        (session
         (capabilities
          (urn:ietf:params:jmap:core
           (max-size-upload 50000000)
           (max-concurrent-upload 8)
           (max-size-request 10000000)
           (max-calls-in-request 32)
           (max-objects-in-get 256)
           (max-objects-in-set 128)
           (collation-algorithms ("i;ascii-numeric"
                                  "i;ascii-casemap"
                                  "i;unicode-casemap")))
          (urn:ietf:params:jmap:mail)
          (urn:ietf:params:jmap:contacts)
          (https://example.com/apis/foobar
           (max-foos-finagled 42)))
         (accounts
          (A13824
           (name "john@example.com")
           (is-personal true)
           (is-read-only false)
           (account-capabilities
            (urn:ietf:params:jmap:mail
             (max-mailboxes-per-email nil)
             (max-mailbox-depth 10))
            (urn:ietf:params:jmap:contacts
             …)))
          (A97813
           (name "jane@example.com")
           (is-personal false)
           (is-read-only true)
           (account-capabilities
            (urn:ietf:params:jmap:mail
             (max-mailboxes-per-email 1)
             (max-mailbox-depth 10)))))
         (primary-accounts
          (urn:ietf:params:jmap:mail A13824)
          (urn:ietf:params:jmap:contacts A13824))
         (username "john@example.com")
         (api-url "https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}")
         (upload-url "https://jmap.example.com/upload/{accountId}/")
         (event-source-url "https://jmap.example.com/eventsource/?types={types}&closeafter={closeafter}&ping={ping}")
         (state "75128aab4b1b"))
    

Or the binary encoding:
{KDc6c2Vzc2lvbigxMjpjYXBhYmlsaXRpZXMoMjU6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29y
ZSgxNTptYXgtc2l6ZS11cGxvYWQ4OjUwMDAwMDAwKSgyMTptYXgtY29uY3VycmVudC11cGxv
YWQxOjgpKDE2Om1heC1zaXplLXJlcXVlc3Q4OjEwMDAwMDAwKSgyMDptYXgtY2FsbHMtaW4t
cmVxdWVzdDI6MzIpKDE4Om1heC1vYmplY3RzLWluLWdldDM6MjU2KSgxODptYXgtb2JqZWN0
cy1pbi1zZXQzOjEyOCkoMjA6Y29sbGF0aW9uLWFsZ29yaXRobXMoMTU6aTthc2NpaS1udW1l
cmljMTU6aTthc2NpaS1jYXNlbWFwMTc6aTt1bmljb2RlLWNhc2VtYXApKSkoMjU6dXJuOmll
dGY6cGFyYW1zOmptYXA6bWFpbCkoMjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMp
KSgzMTpodHRwczovL2V4YW1wbGUuY29tL2FwaXMvZm9vYmFyKDE3Om1heC1mb29zLWZpbmFn
bGVkMjo0MikpKDg6YWNjb3VudHMoNjpBMTM4MjQoNDpuYW1lMTY6am9obkBleGFtcGxlLmNv
bSkoMTE6aXMtcGVyc29uYWw0OnRydWUpKDEyOmlzLXJlYWQtb25seTU6ZmFsc2UpKDIwOmFj
Y291bnQtY2FwYWJpbGl0aWVzKDI1OnVybjppZXRmOnBhcmFtczpqbWFwOm1haWwoMjM6bWF4
LW1haWxib3hlcy1wZXItZW1haWwzOm5pbCkoMTc6bWF4LW1haWxib3gtZGVwdGgyOjEwKSko
Mjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMzWiYpKSkoNjpBOTc4MTMoNDpuYW1l
MTY6amFuZUBleGFtcGxlLmNvbSkoMTE6aXMtcGVyc29uYWw1OmZhbHNlKSgxMjppcy1yZWFk
LW9ubHk0OnRydWUpKDIwOmFjY291bnQtY2FwYWJpbGl0aWVzKDI1dXJuOmlldGY6cGFyYW1z
OmptYXA6bWFpbCgyMzptYXgtbWFpbGJveGVzLXBlci1lbWFpbDE6MSkoMTc6bWF4LW1haWxi
b3gtZGVwdGgyOjEwKSkpKSkoMTY6cHJpbWFyeS1hY2NvdW50cygyNTp1cm46aWV0ZjpwYXJh
bXM6am1hcDptYWlsNjpBMTM4MjQpKDI5OnVybjppZXRmOnBhcmFtczpqbWFwOmNvbnRhY3Rz
NjpBMTM4MjQpKSg4OnVzZXJuYW1lMTY6am9obgo=}

This is the equivalent of the JSON:

    
    
        {
            "capabilities": {
                "urn:ietf:params:jmap:core": {
                    "maxSizeUpload": 50000000,
                    "maxConcurrentUpload": 8,
                    "maxSizeRequest": 10000000,
                    "maxConcurrentRequest": 8,
                    "maxCallsInRequest": 32,
                    "maxObjectsInGet": 256,
                    "maxObjectsInSet": 128,
                    "collationAlgorithms": [
                        "i;ascii-numeric",
                        "i;ascii-casemap",
                        "i;unicode-casemap"
                    ]
                },
                "urn:ietf:params:jmap:mail": {}
                "urn:ietf:params:jmap:contacts": {},
                "https://example.com/apis/foobar": {
                    "maxFoosFinangled": 42
                }
            },
            "accounts": {
                "A13824": {
                    "name": "john@example.com",
                    "isPersonal": true,
                    "isReadOnly": false,
                    "accountCapabilities": {
                        "urn:ietf:params:jmap:mail": {
                            "maxMailboxesPerEmail": null,
                            "maxMailboxDepth": 10,
                            ...
                        },
                        "urn:ietf:params:jmap:contacts": {
                            ...
                        }
                    }
                },
                "A97813": {
                    "name": "jane@example.com",
                    "isPersonal": false,
                    "isReadOnly": true,
                    "accountCapabilities": {
                        "urn:ietf:params:jmap:mail": {
                            "maxMailboxesPerEmail": 1,
                            "maxMailboxDepth": 10,
                            ...
                        }
                    }
                }
            },
            "primaryAccounts": {
                "urn:ietf:params:jmap:mail": "A13824",
                "urn:ietf:params:jmap:contacts": "A13824"
            },
            "username": "john@example.com",
            "apiUrl": "https://jmap.example.com/api/",
            "downloadUrl": "https://jmap.example.com
               /download/{accountId}/{blobId}/{name}?accept={type}",
            "uploadUrl": "https://jmap.example.com/upload/{accountId}/",
            "eventSourceUrl": "https://jmap.example.com
               /eventsource/?types={types}&closeafter={closeafter}&ping={ping}",
            "state": "75128aab4b1b"
        }
    

I think it's pretty obvious which one is cleaner …

~~~
shawnz
> I think it's pretty obvious which one is cleaner …

Yes, clearly it's the JSON. How do you tell at a glance in the s-expr
formatted data what's an object and what's an array? What's a keyword and
what's a literal? I can barely even tell the heirarchy where it's very
visually obvious with the JSON.

~~~
frenchyatwork
I have no strong love for s-expressions, but objects and arrays are
programming language constructs, not data format constructs. There's also no
way in JSON to distinguish between an int and a float and a decimal type, but
that's fine because it's a data format, and not a programming language
literal.

The JSON and s-expr have a different indent level, which is why the hierarchy
is less obvious to you.

~~~
shawnz
> objects and arrays are programming language constructs, not data format
> constructs.

Debatable. To me there is a clear difference between a list of things of the
same type, and a mapping of names to values of potentially varying types. I
don't think the specific programming language really matters, they are two
totally different concepts with much different applications.

> There's also no way in JSON to distinguish between an int and a float and a
> decimal type

True, but they are just small literals which are easy to differentiate at a
glance, not big multi-line structures like objects or lists. Plus
s-expressions have the same issue.

------
sctb
Recent discussions about JMAP in general:

[https://news.ycombinator.com/item?id=19839104](https://news.ycombinator.com/item?id=19839104)

[https://news.ycombinator.com/item?id=18996200](https://news.ycombinator.com/item?id=18996200)

[https://news.ycombinator.com/item?id=18766709](https://news.ycombinator.com/item?id=18766709)

------
felixfbecker
Would love to read blog posts about how to use the API to e.g. manage calendar
events, and then build some CLI tools around that!

~~~
nmjenkins
The JMAP Calendar spec is the next one being standardised. There's a first
draft at [https://tools.ietf.org/html/draft-ietf-jmap-
calendars-00](https://tools.ietf.org/html/draft-ietf-jmap-calendars-00) but
quite a lot more to be added still. Expect a final standard probably first
half of next year.

------
dragonwriter
Is there an extensible JMAP core (what this RFC presents) server framework
available? I know there are JMAP mail/contacts/calendar server
implementations, but it'd be interesting to have a fairly clean starting point
for _other_ potential applications...

------
beefhash
I'm really not sure what I feel about the naming. It's one letter off (both on
the keyboard and alphabetically) from IMAP. Is this an attempt to coattail the
widespread recognition of IMAP or just an extremely unfortunate name
collision?

~~~
jcranmer
JMAP is pretty explicitly meant to replace IMAP.

IMAP was originally conceived essentially to allow a very thin client that
proxies every user command back to the user store, and it also embeds some
reliance on old implementation artifacts (UIDVALIDITY, anyone?). There are a
lot of "fun" corner cases--I once found a message that was interpreted 5
different ways by 4 different implementations. (It doesn't help that MIME
structure is actually pretty loopy and maps _very_ poorly to how all modern
clients view mail messages, resulting in many interesting interpretations).

Modern email clients are all structured very differently from the "thin
client" perspective--everyone keeps their own copy of the email (metadata)
database locally, and use IMAP essentially as a database synchronization
protocol, but IMAP is pretty bad at synchronization, at least until the
LEMONADE additions came in ~a decade ago. IMAP is especially bad at handling
multiple folders correctly, since it assumes that every folder is independent
of the other ones, and you can only sit and wait for messages on one folder
per connection at a time.

