
JMAP – a better way to email - brongondwana
http://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/
======
dannysu
Great job guys! Things like this together with CalDAV and CardDAV are reasons
why I pay you guys for service.

I just tried the following experiments with my Gmail and FastMail account.
There's a reason why FastMail just feels faster!

    
    
      Try this:
      1) While having the Gmail iOS app open, mark an email as read on the web
      2) See how long it takes for Gmail to reflect the change
      3) Now try with FastMail iOS app, see that it's faster
    
      Try this too:
      1) While having the Gmail iOS app open, delete an email on the web
      2) See that Gmail app never updates until you manually refresh
      3) Now try with FastMail iOS app, see that it happens instantly
    
      How about the other direction?
      1) Read an email on your phone
      2) See that Gmail on the web never updates until manual refresh
      3) Now try with FastMail... Instant update!

~~~
veb
Thank you for this comment. I do wonder why they simply didn't _say_ this at
the beginning of the article. I understood what they were up to purely because
I'm a technical guy, but still I didn't want to actually have to watch the
video -- namely because I'm deaf, and far too often there's no captions to
explain what's going on so I prefer to read (and so do people at work, school,
no sound etc), and it takes a lot less time!

I'm really considering moving from Google Apps to Fastmail... who else has
made this jump? Totally worth it?

~~~
hsshah
Every time I read about Fastmail, I want to make the jump from Google Apps.
However, I don't do it because of Google Drive/Docs/Spreadsheet. Any
recommendation for their replacement?!

~~~
veb
Ha. I'm the same, every time I read about them I want to make the jump - but
I'm not reliant on any of those, except mail fortunately.

I just haven't quite done it yet. From people I've spoken to, everyone praises
Fastmail... seems like we all need that extra push.

~~~
sukh
Less than pleasant first experience. Card failed, reason never communicated
clearly enough. Very slow and archaic support.

~~~
aroman
Archaic?

~~~
sukh
Fax documents and payment information, poorly formatted support replies,
supervisor escalation system after a couple different people over 2 weeks
couldn't address a simple on-boarding issue.

------
HarrietJones
This is good, and I love FastMail, but the one thing that really, really
gladdens my heart with this is that they're pushing it as an open standard.

Thanks FastMail. Sincerely, thanks.

------
plg
Amazing.

Modern, innovative, service-oriented products and a business model based on
customers giving the company money in return for services.

I hope that this is a reminder to startups out there that you don't have to
trick customers into handing over their personal information all the while
pretending to offer your services "for free", in order to gain market share
and be profitable.

The "traditional" business model can be quite successful.

~~~
Touche
I hope you're right but I fear that Fastmail isn't successful in a way that
startup founders are going for. Advancing technology unfortunately isn't a
priority for most founders.

~~~
robn_fastmail
We're fully owned by the staff, with no external funding. We've been nicely
profitable since the beginning. Not huge profits but enough to pay for a nice
office, decent salaries and few gadgets.

So yes, if your goal is to attract massive amounts of venture capital and then
get bought out, don't copy us. If you want to have some fun doing something
you love without having to worry about anything else then this is definitely
the way to do it :)

------
driverdan
Why would you create a new email protocol without an encryption requirement? I
understand trying to fix the existing protocol problems but one of the biggest
is plaintext.

~~~
robn_fastmail
Its expected that you'll run JMAP over HTTPS which takes care of the wire
encryption. For end-to-end encryption, there's a heap of problems that still
aren't solved (key distribution, need for content analysis). There's no reason
JMAP couldn't be extended in the future to support these via additional
properties, attached encrypted payloads, etc but mandating it from the outset
would quickly see the whole protocol dead before it began.

~~~
davexunit
>key distribution GPG key servers?

~~~
robn_fastmail
And how do you trust those? And if I'm a brand new user, how do I get my key
in there?

All the moving parts probably exist, but so far no one has pulled it together
into something that a random person off the street can use.

Its not the problem JMAP is trying to solve.

~~~
uaygsfdbzf
You don't need to trust the keyservers since you verify the fingerprints of
downloaded keys against the fingerprints given to you in person.

The keyserver protocol includes commands to include keys in the keyserver
network. How you send keys depends on the UI of the tool you use, geeks will
do this:

gpg --keyserver <keyserver> \--send-keys <fingerprint>

~~~
robn_fastmail
Right. I'm a little rusty on the details. My point stands - key distribution
is not a solved problem for the average user.

~~~
uaygsfdbzf
It is solved for some specific platforms, GNOME for example:

[https://mail.gnome.org/archives/gnome-announce-
list/2014-Nov...](https://mail.gnome.org/archives/gnome-announce-
list/2014-November/msg00004.html)

The average user doesn't use GNOME or Linux though :(

------
ebabchick
For the well acquainted -- how does this relate to Inbox
([https://www.inboxapp.com/](https://www.inboxapp.com/))? I have not spent
enough time with either to make a comparison

~~~
grinich
Michael from Inbox here— a few key differences, from the horse’s mouth. :)

First of all— I want to say that we’re huge fans of Fastmail and really admire
their team and focus. Rob and Neil actually came by for lunch this summer and
hacked in our office for the day. We’ve discussed JMAP back and forth with
them for a while, and specifically had a long thread when we added calendar
endpoints to the Inbox platform. [1]

The fundamental differences between the Inbox APIs and JMAP are the kinds of
applications built with them, the open source code, and the email providers
supported.

The Inbox APIs are really designed to let developers access email data in non-
traditional ways. Our site reads “Say goodbye to IMAP” but we’re not so much
replacing IMAP, as creating a new protocol which liberates email data to a
higher form. We think about email as “the database of your life” which means
unconventional “clients” and access/manipulation patterns.

That’s one of the reasons we have a more REST-inspired design, whereas JMAP is
closer to RPC. The Inbox API endpoints strive to represent canonical objects
that developers simply treat as resources. JMAP is tailored specifically for
traditional email clients, so it excels at fast operations (like archive ->
reload), but compromises in other areas of simplicity and clarity IMHO.

The other main difference is code. The Inbox sync engine is open source with
about 15k lines of code on GitHub.[2] You can run it yourself right now, and
it works on all providers, including Fastmail! That means your users don’t
need to be on any specific provider or backend. If you don’t want to run it
yourself, you can use our hosted version which also includes support for
enterprise environments that use Microsoft Exchange. [3]

I’ve been lurking on the IETF mailing lists for several years now, and one
thing I learned is that spec documents hardly move the world forward.
Developers need code that is battle tested and they can depend on. What use is
a API spec if you can’t build an app with it?

The JMAP system is a great spec for FastMail’s internal mail service, and it
works to power their own clients, but unless they make some drastic changes to
the code and business, it’s not a platform for building 3rd party mail apps.

As an aside, I totally wish I had more time to write. I envy how prolific the
Fastmail blog is these days. :) Keep up the great work Bron, Rob, Neil, and
the rest!

[0] [https://www.inboxapp.com](https://www.inboxapp.com)

[1]
[https://www.inboxapp.com/docs/api#calendar](https://www.inboxapp.com/docs/api#calendar)

[2] [https://github.com/inboxapp/inbox](https://github.com/inboxapp/inbox)

[3] [https://www.inboxapp.com/features](https://www.inboxapp.com/features)

~~~
brongondwana
Hi Michael,

Thanks for commenting! I agree that our APIs are orthagonal in some ways.

We definitely plan to ship open-source code for everything JMAP. I'm working
on an IMAP<=>JMAP proxy, which I'm hoping to have in time for FOSDEM in
February. I'll be giving a lightning talk about JMAP there.

I tell you what - that writing it taking a lot of my time! Evenings, on the
train to and from work. It's not a pace we can keep up forever.

You guys are doing awesome work. Hope to catch up with you again - and hey, it
will be great if you add a JMAP backend when JMAP is a bit more mature.

Worst case for FastMail, we still have an awesome API for our own backend.
Best case, we help everyone else too.

------
ratsmack
This looks interesting, as long as it is more efficient as a protocol and API.
On another note, the entire email infrastructure need to be redesigned in my
opinion.

~~~
alfiedotwtf
> as long as it is more efficient as a protocol and API

It is :)

------
nona
I hate to be critical considering the considerable work they've put in, but I
don't buy their "JMAP is actually more REST-like than most “RESTful” APIs"
statement. But maybe I'm missing a whole lot.

REST doesn't preclude doing multiple things in one round trip. Well it does
seem JMAP is a bit more flexible when it comes to multiplexing very different
actions. But the error handling doesn't seem to tell you which command has
failed. You still can't get away from certain ordering issues. And to be
honest, I don't see how it's going to be that easy to cache.

This really is RPC – using it as a straight replacement for IMAP might work
well, but I wouldn't want to use it as a platform to build something very
different than an email client on it.

~~~
nmjenkins
Each call is tagged by the client and the same tag is added to the response by
the server, so you can tell exactly what error corresponds to which call.

Caching is pretty easy: you just keep a cache of each object type. The delta
update mechanism means you can very accurately invalidate what you need to
keep it up to date. The system is actually really flexible and a great fit for
any CRUD based app; JMAP is really just a combination of a very powerful and
efficient database access protocol, and the definition of some objects to
represent email, contacts and calendars.

~~~
icebraining
Not just some objects, but different methods to access different objects, and
it's those methods that implement the semantics - the base of JMAP is just an
encoding for bulk RPC.

What I mean is, if tomorrow I add a file with the mediatype
application/vnd.icebraining to my server, an intermediate HTTP proxy can still
cache it even though it knows nothing about my format.

But if I add a new object (say, Note) to my JMAP server, along with the
appropriate methods (getNotes, getNoteUpdates, etc), an intermediate JMAP
server will be unable to cache them, since it needs to know the semantics of
the new methods.

So frankly, I don't see how their protocol follows an uniform interface at
all.

I still think it's pretty neat, though, don't take me wrong. I just want
people to stop abusing REST. There's nothing wrong with RPC!

~~~
nona
While I agree with you that in general there's nothing wrong with RPC, I'm not
sure what the advantage in this case is. You seem to have an idea, though…?

~~~
icebraining
Oh, you'd have to ask the JMAP devs. But I do think they'd have to bend the
constraints pretty hard to have a RESTful interface while keeping the same
advantages w.r.t. efficiency in the number of requests and data transferred.

Like Fielding writes in his dissertation, _" The REST interface is designed to
be efficient for large-grain hypermedia data transfer, optimizing for the
common case of the Web, but resulting in an interface that is not optimal for
other forms of architectural interaction."_

------
thecoffman
I haven't had a chance to review the specifics of the protocol yet, but as a
Fastmail user, I can vouch for the speed of event propagation across multiple
clients. Its lightning fast; noticeably faster than say, Gmail.

------
tinco
Wasn't there a startup recently that pivoted from being a mail client into
building a generalized API for interfacing with mail services like
gmail/yahoo/outlook? I forgot the name but I'm super interested in efforts
like this.

I have the feeling the next big social network disruption is going to be
leveraging e-mail in a big way. E-mail is the established quasi-p2p (it's a
network of centralized services) platform that everyone has an account and a
very complete social network on. I'm not saying it would be easy to launch a
Facebook competitor from e-mail, but I believe it's still got the potential to
launch one last social network and someone just has to do it.

I'm hoping the key lies somewhere in building the right sort of mail client,
that's got the reliability and extent of e-mail, adding on top some hook
feature and layering in perhaps a dash of privacy, cryptography and
independence. There's got to be something viral in there.

~~~
aroch
Inbox (MIT and Dropbox):

[http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-
launc...](http://techcrunch.com/2014/07/07/mit-and-dropbox-alums-launch-inbox-
a-next-generation-email-platform/)

[https://www.inboxapp.com/](https://www.inboxapp.com/)

~~~
grinich
We didn't pivot. ;)

------
0942v8653
Did something happen to the comments? I'm seeing 78 upvotes and no comments.

~~~
clooney
Fastmail uses Hacker News to spam, I mean publish all their new developments.

~~~
dvfjsdhgfv
In general I dislike FastMail promotion on HN. But this article is different -
it's some new open development, potentially very useful to many users, and its
success is directly related to the number of developers who will pick it up.
So at least this time it deserves some attention.

------
mortenjorck
This looks fantastic and is a huge, long-overdue breath of fresh air in the
protocol space. But realistically, what will it take for major vendors like
Apple and Google to start adding support for it?

------
hedgehog
Neat. Two questions about the digest:

1\. You recommend using a digest of the message to generate the message ID.
Did you consider mandating that scheme or going farther and using a Merkle
tree for the mailbox representation? It seems like this would allow for
generating single requests that can fetch all new items.

2\. Why SHA-1?

Edit: Also, thanks for keeping Fastmail running. As a customer of about ten
years it's much appreciated.

~~~
brongondwana
No mandated format for message-ID. If you are proxying an existing store with
a stable message-ID, you can just use that. My partially written gmail proxy
uses the MSGID from Gmail, for example.

SHA1 was a good balance between speed and unforgability at the time. There's
still no known collision, and I'm working on putting a piece of randomness
into the LMTP headers that are added to every message on delivery into Cyrus
to make it even safer.

------
mike-cardwell
I understand you guys use and contribute to Cyrus. Is there a JMAP interface
planned for Cyrus? Or for any other mail servers, like Dovecot?

~~~
aroman
from the article itself:

 _" Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon.
No protocol will succeed unless it provides an upgrade path from where we are
now, and a compelling reason to switch. We will provide a proxy which can talk
to existing servers and present them over JMAP."_

~~~
mike-cardwell
Your comment implies that the quote you provided answered my question. It
didn't.

------
maxk42
It's missing the one killer feature that would fix the spam problem: approved
senders. If you had a "friends" list like Facebook where only people you've
approved may send messages to you, we'd cut down on 90% of spam immediately.
Between that and the lack of mandatory encryption this is just another tedious
protocol to implement.

~~~
LeoPanthera
That's not a practical feature in the real world. Sometimes you need email
from unapproved senders. Spam filtering is better accomplished before or after
delivery, not during.

~~~
maxk42
There's nothing preventing an unapproved sender from _requesting_ contact
privileges, but once denied, their messages and further requests would be
ignored by default.

Each server and user would have their own identity management concerns.

Basically someone can send a "friend request" and if approved by a human (the
recipient) they may communicate freely from that point.

~~~
ricardobeat
How is this different from adding an address to your blacklist after the first
message?

~~~
maxk42
They can't deliver their spam message to begin with. You'd need to approve the
contact then you could still block them if they spam you.

There's still friend-request spam on facebook, but not nearly as much as email
because you can just ignore the people you don't know and block people you do
know if they begin to spam you.

~~~
feld
The request for contact would be just as bad as the spam itself...

------
elwell
> By using the platform push channels, JMAP avoids having to hold its own
> connection open.

Can anyone explain what they mean by "platform push channels"? I don't
understand how you can have realtime updates without leaving a connection open
or periodically polling. Do they mean that they use the same mode of transport
as Push Notifications?

~~~
robn_fastmail
Android, iOS, (Windows Phone, Blackberry, etc) have services where a server
application can push a small data packet to a specific device. That device
wakes an application in response which can then do things.

We use this right now in our mobile apps. The apps respond by calling back to
FastMail to check for updates.

More info:

[http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-
good/](http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/)

[https://developer.android.com/google/gcm/index.html](https://developer.android.com/google/gcm/index.html)

[https://developer.apple.com/library/ios/documentation/Networ...](https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/ApplePushService.html)

------
iwantagrinder
I' m interested in this from a security perspective. What does this new
protocol offer in terms of better control around what makes it to the inbox?
Would IMAP>JMAP translation before hitting the user give us better ability to
filter out malicious items/spam?

~~~
SwellJoe
I think the only thing this particular protocol should be concerned with is
reliably providing a method for communicating between client and server. Most
spam and malicious mail filtering should happen as early in the process as
possible, hopefully during the initial connection.

There is a a protocol named Sieve (
[http://www.ietf.org/rfc/rfc5228.txt](http://www.ietf.org/rfc/rfc5228.txt) )
which provides for delivery stage filtering rules. It is similar in capability
and usage to procmail, but formalized and somewhat more modern (and not
capable of running arbitrary system commands), and supported by some modern
IMAP/POP servers, like Dovecot. Presumably if JMAP gets integrated into those
servers, Sieve would also be available.

And, I would hope there wouldn't be an IMAP/JMAP translation layer, but
instead servers would implement JMAP directly. Though I guess in the short
term there might be some sort of proxy.

~~~
gecko
Sieve is the native filtering engine used by FastMail, so I'd say it
integrates quite well with JMAP, from my experience. :)

------
msh
I guess this tries to solve the same issue as microsoft active sync, just in a
open specification way.

------
ummjackson
FastMail continue to impress me. Good work folks.

------
sjtrny
[http://xkcd.com/927/](http://xkcd.com/927/)

~~~
brongondwana
My presentation to the Thunderbird folks led with that, and we would have used
it at Inbox Love too if we'd had time, but a 5 minute presentation had to get
straight to the point.

It's a good comic. Good point.

We didn't take this decision lightly. There are significant reasons why all
the existing standards don't do what's needed, and we made sure we built a
technology preview and used it in the real world first - check the video or
try out FastMail yourself.

------
Animats
What does this do that IMAP and SMTP don't do already?

All you need is an IMAP server, accessed from all your devices. The one that
comes with Android isn't bad. Thunderbird works fine on the desktop.

You don't have to use GMail with Android; when you first power up your Android
device, click "Later" when it asks for a Google login. Then delete the Google
One-Time Startup app. Google won't bother you again.

~~~
corford
tldr; version: It reduces the number of network roundtrips (and server
overhead) needed to keep clients in sync (great for mobile) and addresses the
fragmentation issue between the various server & client implementations of the
sprawling IMAP protocol.

By moving to JSON and HTTP it also makes the barrier to entry a lot lower for
client developers and opens up the possibility to improve speed yet further
with GZIP and websockets.

All that while being able to stay compatible with legacy IMAP servers via a
reference proxy provided for free by Fastmail.

Edit: Oops I've just noticed your handle so apologies for assuming you didn't
read the article. I guess you've read it and still don't see the benefit. In
which case would love to know why you think it's a wasted effort (personally,
I think it looks like a welcome refresh).

~~~
Animats
The Ruby version of the IMAP client isn't that bad. Yes, IMAP doesn't use the
current fad for marshaling, JSON. But a few years ago the web crowd would have
wanted XML, and a few years from now they'll probably want some binary
protocol Google comes up with. "Gzip" compression isn't a big win for objects
the size of email messages, anyway.

IMAP servers that don't use a real database are sure to be a mess, but that's
not the fault of the protocol.

I'd rather not have some startup in the middle of the mail chain. They'll
start thinking they own mail.

------
kijin
JMAP is a rather misleading name.

"J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but
rather the use of HTTP, along with recently added HTTP features such as push.
JSON is just the message packing scheme. HTTP is what does all the heavy
lifting.

"M" is misleading because this protocol is designed for a lot more than mail.
It also handles contacts and calendars, and I wouldn't be surprised if
Fastmail made their file storage service accessible through JMAP as well.

What's even worse is that once you come up with a protocol that covers
everything that Fastmail does, everyone and their dog will try to add more
services to the protocol. Instant messaging? Got it. Collaborative editing a
la Google Drive? No problem. And before you know it, the protocol is bloated
as hell and you've basically reinvented HTTP.

Please don't try to do everything. Do one thing, do it well, and put a strict
limit to the scope of the project for the time being. A few more minutes of
battery life on a phone is not worth polluting the world with yet another
example of massive scope creep. IMAP+SMTP using a stateless protocol would be
cool, but anything more and I'm not sure.

~~~
quink
First thing I thought: It's one better than IMAP and meant to replace it. I++
== J. JMAP.

~~~
gboss
You mean ++I == J

