
JMAP: A better way to email (2014) - robert_foss
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/
======
nmjenkins
Just as an update since then: a JMAP working group was chartered at the IETF a
few months ago, and is working towards bringing the spec to RFC status:
[https://datatracker.ietf.org/wg/jmap/about/](https://datatracker.ietf.org/wg/jmap/about/)

~~~
brongondwana
Thanks for posting that, was about to say the same thing. Am one of the co-
chairs. AMA.

------
mike-cardwell
After reading the "Why is JMAP better than IMAP?" section on
[http://jmap.io/](http://jmap.io/), what it seems to boil down to is small
efficiency gains and an easier time for people writing clients. It doesn't
look to me like users of clients will really gain much from their client using
JMAP instead of IMAP. There's a comment about not requiring a persistent
connection being better for mobile clients, but I've been using IMAP push with
K-9 on my phone for a long time now, and I've never noticed a degraded battery
life from using it.

A quick look at the protocol looks like in order to get a list of mailboxes,
you have to ask for them all. In IMAP, you can say, show me top level
mailboxes, show me direct child mailboxes of mailbox x, etc. In a previous
role, I was responsible for working on a mail archiving solution which had a
large hierarchy of folders (into the thousands IIRC) in a single account.
Access to the archive was via IMAP. Looks to me that JMAP here would require
the client to potentially fetch megabytes of data just to start showing a list
of folders, where the solution we had with IMAP probably took a couple of
kilobytes at most to get started.

It does make me think that the protocol may have been designed with less
thought for uncommon use cases than IMAP was, but I've only spent a few
minutes looking at the protocol, so I could be wrong.

~~~
brongondwana
You're very welcome to comment on the IETF mailing list if you think you can
make a compelling argument for having getMailboxes also take a filter allowing
you to build a tree with "parentId": [null] and then use the IDs from each of
those responses to create a "parentIds": [x, y, z] query.

Of course nothing stops you storing thousands of mailboxes in a non-
heirarchical layout and making IMAP have large LIST fetches too.

(I do wonder - did your solution via IMAP still fetch everything each time,
because in that case all you're doing is spreading that megabyte out over
multiple round trips)

~~~
mike-cardwell
> You're very welcome to comment on the IETF mailing list if you think you can
> make a compelling argument for having getMailboxes also take a filter
> allowing you to build a tree with "parentId": [null] and then use the IDs
> from each of those responses to create a "parentIds": [x, y, z] query.

I'm not invested enough to sign up to a mailing list and try to convince
people to change a protocol. I'm only invested enough to fire off a comment on
a forum I am already signed up to. I explained my use case above. Whether or
not it is compelling is up to the reader to decide. Maybe I'm the only person
in the World who has dealt with large hierarchies of mailboxes.

> Of course nothing stops you storing thousands of mailboxes in a non-
> heirarchical layout and making IMAP have large LIST fetches too.

Yes. I would issue the LIST command, and as the IMAP server started streaming
the results to the client, I would immediately process each mailbox as soon as
it was down, rather than waiting for them all to be retrieved. I fear that
with a JSON based protocol, most server and client implementations wouldn't
deal with streaming (although they technically could), and would just deal
with completed blobs.

Does the Cyrus implementation construct the entire blob of JSON before
beginning sending it to the client, or does it start constructing and
streaming the JSON as soon as possible, before it has even fetched the entire
list of mailboxes from whatever the backend data store is? The JMAP streaming
solution seems like a more complicated solution for developers to implement
than the IMAP streaming solution.

> (I do wonder - did your solution via IMAP still fetch everything each time,
> because in that case all you're doing is spreading that megabyte out over
> multiple round trips)

People were accessing this mailbox with many different clients. How each of
those clients worked under the hood, I do not know. I did build a web
frontend, and that web frontend worked efficiently, fetching only the folders
that were being displayed to the user, and their immediate children. A user
clicked a folder to expand it, and they would immediately see the pre-fetched
child folders, and a new request to fetch those folders children was then
triggered.

~~~
brongondwana
Sadly protocols are made by those who show up and put the effort in.

I'll pass a link to this comment on to the list and see if others chime in.
The downside of allowing filtered getMailboxes is more protocol complexity,
but maybe it's worth it.

On to the next topic:

Cyrus creates the entire blob. Even when it's streaming a LIST reply it still
batches a lot in memory. You'll discover pretty fast if you don't do that, you
might wind up holding a lock for a LONG time if you're trying to build
consistent replies and the remote end is on a slow network link or just
consuming slowly. So we batch all replies right now.

If I was building a brand new product I might use an engine which supported
MVCC (like Postgres database for example) and hence have flexibility. Having
said that - even then you might find that a server talking to slow clients
would want to buffer to temp file and free up memory and locks while it
streamed the data.

With HTTP we get around this by having nginx handle all the mess, so I guess
that's what will happen with JMAP too, nginx buffering large bodies to disk.

~~~
oddlyaromatic
>Sadly protocols are made by those who show up

This is a very high burden to place on casual feedback from somebody trying to
be nice.

~~~
brongondwana
I'm not quite sure how to respond to this. The IETF is aware that there's
already a high burden on people, that decisions tend to be made mostly by the
people who can get to meetings and talk in the rooms where standards are
hashed out.

Having said that, I'm not sure I agree that it's an unreasonable burden to ask
somebody to join a mailing list and defend an opinion if they want to
influence a standard which will hopefully be useful for tens of years and
implemented by hundreds or thousands of people. Any additional complexity for
server and client implementers becomes quite a lot of work multiplied by all
those people, and the additional burden of implementing a bigger standard may
even be enough to discourage people from using it at all.

And it's a fact, the decisions are made by those who show up and put the work
in. Wishing it was otherwise doesn't change the reality that we need to make
tradeoffs and that the IETF favours rough consensus and working code. Both of
which take work.

I will forward this discussion to the mailing list, that's about all I can do
as chair, I won't proxy all the IETF work over to comments on HN!

~~~
oddlyaromatic
That makes sense. I think you jumped to quickly to interpreting the original
comment as an intent to influence the standard. I read it as just "here is a
thought that might be useful/relevant to the people who are making the thing
so that they can use it as they see fit when making the standard".

No big deal either way!

------
shmerl
Does this trend of replacing TCP/IP with HTTP start being more common, or it
just looks to me that way? I get a feeling that HTTP is stretched too much
outside of what it's designed for, and it results in messy design.

~~~
vbezhenar
Most important thing is JavaScript from browser can use only HTTP, so if you
want to build web-based mail client, you won't have to build proxy HTTP
service, you can just implement protocol on client. Another important thing is
proxy, many organizations don't allow anything but HTTP, so it's easier to
just use it.

HTTP is not that bad, anyway. You don't have to use every quirk.

~~~
djsumdog
I agree with your comments about web-based e-mail clients, but:

> many organizations don't allow anything but HTTP, so it's easier to just use
> it.

Remember when we allowed for newer protocols to have their own ports and
didn't blacklist everything but HTTP? Why are we doing this again, now? Please
don't say security/firewalls or else I'm going to ram an ice pick in my eye.

If your organisation is blocking ports, they're probably filtering HTTP.
Companies that care about filtering deny access to most personal e-mail on
work machines. So basically you're saying "let's abuse HTTP so people can
violate company policy."

You may or may not also being saying, "Let's build a brand new awesome shiny
latest and greatest protocol...and permanently tie it to the technical debt
wreckage of HTTP/2.

oh and Google could just fix their terrible broken IMAP implementation.

------
tyingq
I found this observation interesting:

 _" Our servers are in New York, and our developers are in Australia. Even on
a good day, the ping times are over 230 milliseconds. Anything more than the
absolute minimum number of round trips is felt very keenly."_

I wonder if some web apps might be built better if developers were bandwidth
throttled and latency hindered artificially on their own machines, for say,
one day a week.

~~~
arkitaip
This is a useful built-in feature of the dev tools on Chrome so for devs who
care about their users it's trivial to use.

I think moving towards progressive web apps (PWA) should make the web much
faster because good performance over poor bandwidths is one of the
cornerstones of PWA.

~~~
tyingq
Yes, this thing in chrome: [https://developers.google.com/web/tools/chrome-
devtools/netw...](https://developers.google.com/web/tools/chrome-
devtools/network-performance/imgs/throttle-selection.png)

PWA should help somewhat, but there is still quite a bit of javascript bloat
and "too many assets" to combat. When my phone runs out of the included 4G and
drops to 3G, many sites are unbearably slow.

------
brongondwana
Thanks HN - while I'm away at Mt Gambier supporting my kid in
[http://generationsinjazz.com.au/](http://generationsinjazz.com.au/) you post
not one, but TWO things about my hobby horses. Turns out 6000 people in a
field tends to overload the local cell, so I've had very spotty coverage.

Sitting in a bus with unreliable phone connection on my way back home trying
to reply to everything now. Cheers.

~~~
jorangreef
"Turns out 6000 people in a field tends to overload the local cell"

Next time, try dropping down to 2G and you should make it through fine.

~~~
brongondwana
Good idea :) Though putting the phone away and cheering on our kids was pretty
worthwhile too.

------
mostafah
Email protocols are very old and network APIs have improved _a lot_ during the
last two decades. So, when FastMail (I’m a happy customer) introduced JMAP I
was excited about it. But even then I didn’t anticipate much success for it.
Big email providers (Gmail, iCloud, Outlook, …) don’t care about these things.
And it’s a shame that email providers and clients are still wrestling with
ancient protocols to provide a decent experience for saving drafts, search,
folders/labels, and other basic things.

~~~
StavrosK
I hope it gains more traction. In the mean time, I've switched, to Fastmail,
and I couldn't be happier. It really is much faster than Gmail, I was very
surprised because I didn't think email could be so fast. I guess Gmail had
become the new normal.

~~~
jpgvm
Except for search really. I love FastMail but they need to work on the search
speed. Atleast it's faster than Outlook.com though, that search speed is
beyond unacceptable.

~~~
brongondwana
If you open a support ticket I can look at your search issues, ask for Bron.
It shouldn't be slow unless you're deliberately bypassing the Xapian indexes
by using substr searches.

~~~
Semaphor
How long is search supposed to take? Entering something common (like "gmail")
takes around 5 seconds for me with 32k results. My old google mail account
takes 1.5s to display the first page.

~~~
mike-cardwell
Looking at the Cyrus code, the server doesn't even start sending search
results to the client until it has completed running the search. And due to
the response being JSON, and there being no pagination in the protocol,
presumably the Fastmail client wont start displaying the results until it has
retrieved them all either. So I'm not surprised it's slow.

~~~
brongondwana
Are you looking at JMAP getMessageList here? It paginates the response
(offset/anchor and count), but yes - it has to build the complete list. Not
that we're using that in FastMail production yet, we're using XCONVMULTISORT
still. It's roughly the same algorithm though.

The biggest expense is all the index.c stuff to support IMAP protocol
requirements. There's a lot of cost in there which will go away with JMAP, and
which will go away even more when we replace the separate cyrus.index files
with a structured database per user. That's still got a ton of work to do
though... one of the goals for long term Cyrus improvements.

~~~
mike-cardwell
Does this allow for the client to say, "Give me all results matching x", and
for the server to say, here's the first N results, but there are potentially
more, and then the client can then ask for more, in multiple requests until it
has them all, allowing them to progressively show results until they're all
retrieved?

That's how I would expect a JSON based protocol to work for pretty much all
types of requests that provide potentially large lists in a response. I would
expect getMailboxes to work this way too to be honest.

------
LeoPanthera
More recent info at [http://jmap.io](http://jmap.io)

------
Bino
Why dont improve on IMAP with extensions, I can't see how the "extending IMAP
only makes it worse" argument hold. How would adding an addition protocol
(JMAP) to a buggy client make the situation any better? I can see how it's
easier for fastmail as a company to try to get its changes out by making
another protocol, but would we not benefit better by trying to improve on
IMAP? it's very unlikely this will ever replace IMAP and SMTPS from history
internet always try to iterate and improve not replace standards completely?
There is great effort to improve on SMTP and encryption lately, what can we
learn from that?

------
irl_
If you want to try out JMAP on a provider that is not FastMail, you use a
proxy:

[https://proxy.jmap.io/](https://proxy.jmap.io/)

I've not seen it deployed anywhere else yet as a production thing, and I don't
know how much progress has been made on standardisation of it, but it does
seem to offer vast improvements over IMAP.

~~~
lemming
Do Fastmail themselves let you use jmap to connect now? They didn't a while
back but I haven't checked recently.

~~~
brongondwana
Not yet, though it's being used internally for a few things already.

The ietf working group and the Cyrus IMAP source code are the most up to date
right now. I'll be updating the proxy (perl, open source) soon.

------
im0nde
What about encrypting/signing? Does it address that too?

~~~
brongondwana
Explicitly not. Key management is a far from solved problem. We're not trying
to fix everything.

------
Mojah
Happy to see this get more attention again. I highlighted it in last week's
cron.weekly [1] too.

I'm still unsure of the viability at this point, but time will tell.

[1]
[https://www.cronweekly.com/issue-78/](https://www.cronweekly.com/issue-78/)

