
JSON Mail Access Protocol Specification (JMAP) - alfiejohn_
http://jmap.io/
======
bodyfour
Unfortunately it includes my biggest gripe about IMAP -- the requirement that
messages are given a server-side message number. Maybe breaking with IMAP on
this would make interop too hard though.

This requirement is convenient for the client but makes implementing robust
servers harder than it need be. If instead each message was identified by a
{timestamp,UUID} tuple then multiple MX servers could do final delivery to an
eventually-consistent shared store. The requirement for strict, durable
message ordering is the one thing that forces hard synchronization among
multiple machines doing delivery.

POP3 actually made a step this direction with the common "UIDL" extension;
IMAP4 felt like going backwards on this.

While I'm on the subject, another less-common POP3 extension I liked was "XTND
XMIT". This gave a simple way for the POP3 client to send mail; sadly it never
really caught on in favor of just using SMTP submission. Originally though
this only worked OK because SMTP was open-relay. Once spam put an end to that
it became complicated to send a message when accessing your mail remotely.
Then to solve _that_ problem we had to graft an auth layer on top of the
existing SMTP protocol and upgrade all of the clients and servers to support
it. If only "XTND XMIT" had won in the beginning everybody would be submitting
outgoing mail over that already-authenticated channel and so much work could
have been avoided.

So anyway, those are my two requests for any "IMAP killer": allow the server
to use unique (but non-sequential) IDs for messages and provide a way to
submit outgoing messages over the same channel.

~~~
masklinn
> Unfortunately it includes my biggest gripe about IMAP -- the requirement
> that messages are given a server-side message number. Maybe breaking with
> IMAP on this would make interop too hard though.

I don't understand this criticism, the criticism is usually that IMAP does
_not_ require maintaining UIDs (the server can announce UIDNOTSTICKY[0])

> While I'm on the subject, another less-common POP3 extension I liked was
> "XTND XMIT". This gave a simple way for the POP3 client to send mail

[https://news.ycombinator.com/item?id=4826447](https://news.ycombinator.com/item?id=4826447)

> IMAP has actually been extended to support that feature (not just kind of
> randomly, but by the IETF "Lemonade" working group), even allowing (in
> concert with other extensions) compositing a message from parts (such as
> attachments) on the server without having to download them first to the
> client.

> However, and I think this is actually more relevant in this context, Mark
> Crispin was an "opponent". (Note: the following e-mail snippet by Mark
> Crispin, was written years after the Lemonade Submit proposal expired, and
> thereby should be considered to already take into account that context.)

>> For many years, there have been various proposals to add mail sending
capabilities to mail access protocols such as POP and IMAP.

>> These proposals are always strongly opposed. It is one of the "attractive
nuisances" of email protocols. The value of the capability is obvious to many
people, but the high cost of having it in POP or IMAP is much less obvious.

>> I am one of the opponents. For the past 25 or so years we have been in the
overwhelming majority. It is quite unlikely that this concensus will change.
If anything, it has become stronger in recent years.

>> […]

>> If you're not really that curious, suffice it to say that the people who
design the protocols and systems understand the attraction but have excellent
reasons not to do it.

[http://mailman2.u.washington.edu/pipermail/imap-
uw/2009-Janu...](http://mailman2.u.washington.edu/pipermail/imap-
uw/2009-January/002345.html)

[0] [http://mailman2.u.washington.edu/pipermail/imap-
protocol/200...](http://mailman2.u.washington.edu/pipermail/imap-
protocol/2007-March/000436.html)

~~~
nmjenkins
(Author of the spec here)

We will be adding sending to the JMAP spec, although support may be optional.
The way it currently works at FastMail is a bit too specific to our
architecture though, so we want to clean it up before adding it to the spec.

~~~
masklinn
>>> These proposals are always strongly opposed. It is one of the "attractive
nuisances" of email protocols. The value of the capability is obvious to many
people, but the high cost of having it in POP or IMAP is much less obvious.

~~~
nmjenkins
Lucky I'm not considering adding it to either POP or IMAP then. :)

~~~
masklinn
Good luck with your doom-repeating experiment, I guess.

------
cromwellian
HTTP itself has an overhead, I'm skeptical changing a persistent IMAP
connection into a request/response model with a different serialization format
is going to be more efficient, especially given compression (like
COMPRESS=DEFLATE)

IMAP4rev1 is one thing, but there are many extensions supported in modern imap
servers to speed up syncing and mailing, like
CATENATE/BURL/CONDSTORE/MULTIAPPEND/MOVE/etc

Granted, the IMAP protocol is pretty hairy and difficult to work with. On the
other hand, there's a huge ginormous deployment of it on the client and
server, and the IETF WG behind it. I doubt JMAP will replace it anytime soon
unless the WG itself takes up the issue. And given the neckbeards there ;-),
some of whom have been working in IMAP for decades, you face the same
opposition to change as those trying to legalize drugs or gay marriage. ;-)

We're still trying to get IPv6 deployed!

~~~
nmjenkins
One of the main efficiency gains is in massively reducing the number of round
trips required to do a series of operations. This batching of operations is
hugely important on high-latency connections, relevant for us in Australia all
the time but also of importance on mobile connections the world over. Even 4G
networks have relatively high latency, but can do a burst transfer pretty
efficiently.

~~~
jorangreef
A binary protocol running over persistent socket would have fewer round trips
and parsing latency. HTTP and SPDY can't compete with that, because they
introduce too much header overhead and complexity.

~~~
robn_fastmail
Persistent connections are tough on patchy mobile networks.

Parsing latency isn't really an issue; all this stuff is IO bound anyway. CPU
cycles are cheap.

~~~
jorangreef
If persistent connections are patchy, then HTTP is no further ahead.

A sure-fire way to decrease latency is to send as little as possible.

An empty default browser HTTP request is likely to cost you around 500 bytes
in headers, before you have added any headers or data. Contrast that with a
binary command which can be 1 byte. And then multiply this by hundreds of
requests. CPU cycles on mobile are not cheap. It makes no sense to parse 500
bytes (and then GC this later) when you don't have to.

~~~
MichaelGG
Even on servers, parsing isn't cheap. Look at nginx code or other high-
performance parsers. All sorts of little tricks to compare 4 or 8 bytes at a
time to determine the verb and whatnot.

------
brongondwana
This is similar to what we're using at FastMail for our web interface, but
simplified and made a bit more generic based on our experiences over the past
year.

Our plan is to port our web interface across to this spec and implement both a
direct backed in Cyrus IMAPd and a standalone implementation which can proxy
for existing IMAP servers.

~~~
kijin
Keep up the good work! 6-year customer here. Notwithstanding some lost
features compared to the old UI, your new web interface really lives up to
your company's name. Everything is fast and snappy even when I'm accessing it
from halfway around the world.

But I'm not sure it's a good idea to emphasize the fact that you use JSON.
Sure, it's cool right now, but I find it technically more interesting that it
replaces IMAP with something that is built on top of HTTP, a somewhat RESTful
protocol.

~~~
brongondwana
You need some form of structure unless you want a zillion roundtrips. JSON is
cool because it provides the basic structures (scalar, list, map) in a
consistent and reliably parsable form. You don't need a context-aware 50 page
ABNF to be able to parse it cleanly.

(don't even get me started on trying to write a proxy for IMAP that's aware of
the connection state)

------
micheljansen
Interesting step, but the decisions taken around concurrency seem to assume
small-scale, single-user use cases only. I would be the first to admit I
haven't given this enough thought yet, but to me it seems that a global lock
and MODSEQ make distributed usage etc. very difficult. Isn't this a step
backwards?

~~~
nmjenkins
No. See my answer to jorangreef above, but essentially this protocol is for
server <-> client sync, but you can still do efficient server <-> server sync
for distributed backends.

------
jmspring
Call me curmudgeonly, why replace a protocol that works...instead roll up a
library to provide the functionality...

Those things called standards generally find their way through bodies like the
IETF. Maybe I missed it while reading on my phone, but I don't see this
submitted as an IETF draft...

~~~
alfiejohn_
Because AFAIR, you need to point to two working implementations of a spec
before turning it into an RFC.

~~~
teddyh
Well, no, first you discuss the idea in a Working Group mailing list, which is
like HN – you get lots of great negative feedback. When you have fixed all
your initial bugs, you write a _draft_ RFC. With which implementers can create
independent implementations. If there are not anyone willing to do this, this
is a sign that the idea is not wanted enough, i.e. not widely useful enough to
warrant an RFC in the first place. The implementers will find bugs. You fix
those, and discuss some more. Eventually, you, the mailing list and the
implementers will be happy with the standard, at which point you move to have
the RFC published as an official RFC.

Anyhow, this is all for a “Standard track” RFC. An “Informational” or
“Experimental” RFC has no such requirement, and can basically be submitted by
anyone for anything.

------
advisedwang
Does JMAP have an equivalent to IMAP IDLE? I.e. the ability for the server to
notify the client when an email arrives. This means client's don't need to
poll, and messages are seen instantly.

For me instant notification of inbound message is a must-have.

~~~
brongondwana
We're using eventsource push - just to say "you need to poll for changes now".
I think that's cleaner than trying to push the actual changes.

Our actual push does include some extra detail, but the important bit is just
"here's a new state that exists on the server - if you haven't already heard
of it, you probably should poll for changes now".

It's out of band(ish) - because JMAP is connectionless. And because depending
on which platform you're using, there might be a nice channel for push
notifications already.

~~~
jkarneges
How about documenting the EventSource interactions?

~~~
nmjenkins
We probably will publish this as well. We're focussing on the core API first.
Having said that, it's really very simple. Here's the entirety of our
(currently internal) spec for push events at FM:

This is a text/event-stream resource, as described in
[[http://www.w3.org/TR/eventsource/]()](http://www.w3.org/TR/eventsource/\]\(\)).
The following events are pushed:

\- __progress __: Sent during a long-running api method call to let the client
know not to timeout the connection. The data is a JSON object with a single
property: `connectionId`, with the id the client sent for that connection.
This event does not set a new id in the event source stream.

\- __push __: Sent whenever the user 's highest modseq changes. The event id
is the new highest modseq. The data for the event is a JSON object, with the
following properties:

    
    
      * **clientId**: `String` (optional). If the change was due to an action initiated by
        the API, the `X-ME-ClientId` header will be echoed back as this property. This
        allows clients to ignore push events for changes they themselves have made.
    
      * **mailModSeq**: `Number` (The highest modseq for a mailbox)
    
      * **contactsModSeq**: `Number` (The highest modseq for contacts/contact groups)
    
      * **calendarModSeq**: `Number` (The highest modseq for calendar events).

------
mercurial
Interesting. What is the reason for grouping method calls together? Is this to
reduce the number of requests sent to the server? If not, is there no way of
providing a higher level operation instead of bundled discrete operations?

In any case, I think you could have something a lot closer to REST, even if
you carry on with grouping calls together.

Something like:

    
    
        [
            [ "GET", "/messages", { "search": "foo" }],
            [ "GET", "/mailboxes", { "etag": "bar" }]
        ]
    

Obviously, you're not limited to HTTP verbs if you do it like this, but it's a
familiar metaphor.

~~~
nmjenkins
(Author of the spec here)

REST is way too slow. The JMAP format massively reduces round trip times,
especially when you have sequential operations where you must wait for one to
finish before the next can happen. And once you're not doing REST, I don't
personally think putting HTTP verbs in the method calls doesn't really make it
any clearer, and is just likely to confuse matters.

~~~
mercurial
> The JMAP format massively reduces round trip times, especially when you have
> sequential operations where you must wait for one to finish before the next
> can happen.

I hear you (REST and batch operations don't mix too well). But wouldn't it be
possible to simply provide higher level operations, grouping several of these
commands?

> And once you're not doing REST, I don't personally think putting HTTP verbs
> in the method calls doesn't really make it any clearer, and is just likely
> to confuse matters.

As I said, it wouldn't necessarily be restricted to HTTP verbs (eg, using
'create' and 'update' instead of 'POST' and 'PUT'), but I do like the idea of
separating the resource you act on from the action on this resource.

------
bjourne
IMAP is mostly slow as hell when dealing with large mailboxes. For example
fetching the 10 most recently updated threads (for a sane definition of what a
thread/conversation is, which imap doesn't deal with at all) is insanely slow.
It entails scanning the whole mailbox trying to figure out what the threads
are and when someone last posted to them. Not fast when you can have millions
of mails in a mailbox. Is that the kind of problems JMAP might solve?

Also interested in hearing if you know anything about GMail's json api? Surely
they must have something similar to JMAP which their web client is using even
if it the api is proprietary?

------
cursork
I'm a big proponent of everything-behind-http because of the way it introduces
a common language for integrating disparate services.

However! as someone who has had to debug some particularly gnarly email issues
in the past I really do quite like the ability to telnet into an IMAP server
to dig around. I also have an appreciation of some of the more horrific things
that can go wrong...

So my point is that I'm not sure I agree with "generally much easier to work
with than IMAP" \- there's two types of "working with". One is building a new
client/server from scratch - I'd agree this might be easier, but not something
that is done _that_ often. The other is being able to quickly investigate
issues with hand-crafted queries, and unfortunately I'd say a simple plain
text question-answer interface is better - and here IMAP wins.

Maybe I'm just getting old - it's probably true that almost everyone is
happier with HTTP than "telnet-able" protocols (SMTP, IMAP, Redis, etc.)
nowadays.

------
jbb555
Lets reinvent everything badly in javascript.

------
tedivm
I maintain an imap client library for PHP (
[https://github.com/tedivm/Fetch](https://github.com/tedivm/Fetch) ), which is
far more painful than it really should be. I really am happy to see attempts
at a more modern approach.

------
torrance
As a complete aside, what is the language used under the section
"getMessageList"?

~~~
ivank
That's just their own pseudocode language

------
jorangreef
Any new access protocol for mail should be designed to work over persistent
secure socket, not over HTTP. Additionally, there should be a minimum of work
imposed on the server by the client.

~~~
alfiejohn_
See the comments above about mobile networks and persistent sockets.

------
compare
I swear I was also thinking of inventing a JSON-based replacement for the IMAP
just last week.

------
eitany
Looks very useful.

