

An open source JMAP proxy, JavaScript library and webmail demo - cdvonstinkpot
http://blog.fastmail.com/2015/08/11/an-open-source-jmap-proxy-javascript-library-and-webmail-demo/

======
atonse
Can the JMAP protocol be made easier with HTTP/2? The protocol has its own
sort of multiplexing (multiple calls in one HTTP call, worrying about
executing things in a certain sequence, etc)

Wouldn't an HTTP/2 transport mechanism remove the need to do all this and
greatly simplify it to feel more RESTy, but get all the benefits of multiple
calls with near zero overhead?

~~~
robn_fastmail
JMAP itself is transport-agnostic, though of course we've only implemented it
over regular HTTP so far.

Some methods have to be batched together and executed in order, as it's
possible to arrange for the output of one method to be used for an input to
another method within the same batch. It is however entirely feasible to have
multiple batches containing a single method in-flight at the same time and
indeed, the demo client and the FastMail client do just this.

I'd certainly be interested to see if HTTP/2 could help anywhere but I don't
think there's many obvious wins - JMAP itself has very little overhead.

------
tracker1
I really hope that this or something similar sees some uptake... Although I'm
not familiar with JMAP I'm relatively familiar with SMTP, POP3 and IMAP and to
be honest, they kind of suck and support things not needed, and things that
are seem hacky...

A new interface for mail, contacts, calendar has been sorely needed for a
while.

~~~
rcarmo
I, for one, am somewhat leery of people reinventing the wheel - especially for
a browser-based environment, where everything has a history of being brittle
and short-lived.

IMAP works amazingly well if you're using dovecot as a server (including
search and push e-mail), and CalDAV/CardDAV also work well if your server
actually implements things properly, and projects like Horde have shown you
can run and deploy web-based services for them at scale (even though I still
regret they're using PHP).

Sure, anything related to DAV has more than a few warts, but my main point
here is that things like JMAP won't cater to an installed base or deliver
benefits across the board until they finish filing all the corners on their
shiny new... cube? Dodecahedron?

~~~
robn_fastmail
> I, for one, am somewhat leery of people reinventing the wheel - especially
> for a browser-based environment, where everything has a history of being
> brittle and short-lived.

Yeah, so are we (and we even introduce JMAP with the "now you have X
standards" XKCD comic). This is our last resort. We've been pushing the limits
of what's possible with IMAP for years, and we've come to the conclusion that
there's just some things you cannot do efficiently with it.

JMAP is not not just for browsers, though that has been where our focus is so
far. I have partial implementations of JMAP for fetchmail and mutt already
done (I'm just trying to find time to finish them and get them out there).
Yes, JMAP uses JSON for its data encoding and (currently) HTTP for its
transport, but it's not inherently browser-based.

> IMAP works amazingly well if you're using dovecot as a server (including
> search and push e-mail), and CalDAV/CardDAV also work well if your server
> actually implements things properly, and projects like Horde have shown you
> can run and deploy web-based services for them at scale (even though I still
> regret they're using PHP).

The problem is, if you're not using a server like Dovecot or Cyrus that does
implement all the extensions to IMAP that you need to get a modern email
experience, then you're screwed. Ditto if your client doesn't support it. So
we end up in the situation where we have all this stuff that is technically
IMAP-compliant (ie implements RFC 3501) but gives a half-assed experience to
the user.

Worse, if you wake up one morning and decide you want to build some amazing
new experience on top of standards-compliant email, you'll spend the next few
months trying to get your mail client bits working correctly before probably
giving up. Even if you do get it working, you'll still have to work out what
to do against servers that don't support CONDSTORE and QRESYNC. This is why
all these pretty new mail-based apps generally target Gmail exclusively - it's
a known environment.

So that's what JMAP is about. Use it and you get a straightforward protocol
based on tech you probably already know that supports things you expect in
2015, like push, assistance on flaky networks, battery friendly, etc.

> Sure, anything related to DAV has more than a few warts, but my main point
> here is that things like JMAP won't cater to an installed base or deliver
> benefits across the board until they finish filing all the corners.

Do you see anything missing from the spec?

We recognise that this doesn't work without everyone supporting it, which is
part of why we're releasing the spec and all this code and talking to every
company with an email product that will listen. This won't work unless
everyone can play and its easy for them to do so. We don't want to own this;
that doesn't help anyone.

I don't know if JMAP will win. Something has to change though, that I'm
convinced of. The web had its revolution (a couple of times). We think its
email's turn.

~~~
Arnt
When I read the spec the push support was ungood.

What it should do: Tell the server an XMPP or HTTP endpoint to which the
server should push a specificed opaque blob when new mail arrives, and
possibly other opaque blobs corresponding to other changes, and a (non-
binding) rate limit.

With that, it would be easy to write a low-overhead server to support a
particular mobile client. Accepts opaque blobs, decodes a little, sends blob
to phone via giantmobilevendor.com (which also uses XMPP/HTTP), needs to store
no state.

~~~
robn_fastmail
That's pretty much exactly what it does. Client supplies a HTTP endpoint, that
gets called with new state strings. That endpoint arranges for that
information to be shipped to the client and then the client issues calls to
update methods to get its view of the world up to date.

Its easy to extend to a custom (non-URL) method type if you like, and works
well with mobile push channels. This is what we do with our mobile apps at
FastMail to push through GCM, APNS, ADM and Pushy.

~~~
Arnt
Glad to hear you've fixed that, then.

------
bengotow
Got sidetracked reading through the Overture.js documentation and source,
which I hadn't seen before. It seems like a really impressive Javascript
framework. Live queries ie. data views, lots of great animation support, nice
set of stock components... It's a real shame they put all this work into a
monolithic framework rather than contributing packages / ideas / insight to
something like React or the broader ecosystem. I feel like these kitchen-sink
frameworks are more or less on the way out -- this is probably a fantastic
body of work that few people (myself included!) will learn well enough to
appreciate.

~~~
nmjenkins
Well, Overture was created several years before React came along (it actually
predates Angular and Ember as well), so it would have been hard to make it as
plugins to this! It's not actually as monolithic as it first appears: a small
change to the build tool could easily let you just build the parts you want
(e.g. just the view system, or just the datastore). Each folder inside the
source/Overture/ directory is really a separate module. The dependency
information between the modules is already in the files so that the build tool
orders them correctly, so it would be a small extension to make it only
include the modules you want + dependencies.

Basically, the library is perfect for us (and we think it's pretty neat; our
web app is one of the largest, most complex and fastest out there, and the
codebase is very clean). But we totally understand that the lack of more
documentation, examples (or even books, tutorials and seminars!) compared to
the popular libraries makes it harder for people to get into. We still wanted
to put the code out there in case someone found it useful, even if only as
inspiration rather than using it directly.

If you want to see an example of idiomatic Overture usage, take a look at the
source of the JMAP demo webmail. It's fewer than 2000 lines of non-library
code, and yet packs a lot in!

------
616c
I remember brong and friends from FM talking about this on blogs and maybe the
Email Forum. And it is finally starting to land!

Very cool. Gotta love FM. Thanks from a paying customer!

------
lukeh
What's the authentication story, is it done at the HTTP level?

~~~
robn_fastmail
Yes, via the Authorization: header with a negotiated access token.

The spec includes a discovery and authentication mechanism that a client can
use to get an access token and can work fine with passwords, OAuth, etc. We
don't actually have a lot of experience with this part of the protocol though.
We expect that vendors may want to provide an alternate method to obtain
access tokens (maybe using OAuth tokens directly). We're expecting to learn
more about these use cases as more implementations appear.

~~~
gdamjan1
I hope it works with 2 factor auth (like TOTP) for applications too. Perhaps
it's a bit outside of the protocol scope, but a "best practice" suggestion
would be great.

~~~
robn_fastmail
I don't believe there's anything in it that would prevent that. If you squint,
the auth exchange looks a lot like SASL, and that supports pretty much
anything.

This is something we'll gain more experience with at FastMail over the coming
year. We'll be sure to share that experience.

------
gdamjan1
Are there any other implementations of JMAP, especially for calendar/contacts?
Also, android sync service for the same?

~~~
robn_fastmail
There's not much around yet. The only things that are in any kind of
usable/complete state are the proxy, the demo client and the JS client
library, all mentioned in the original post.

Cyrus has partial support (the basic data model and contacts) and is running
in production at FastMail. The aim is for full support this year.

On the client side, Roundcube has committed to adding support in the next
version (after their recent successful Indiegogo campaign) and Thunderbird is
serious investigating it (there's a GSoC student looking at it).

I'm aware of two companies that will be shipping products (client and server)
based on JMAP this year. I'm not going to mention names because that's not
cool.

Finally, I'm personally building a JMAP data model library in Rust, a
standalone (non-IMAP) server on top of it, and I'm part way through adding
support to fetchmail and mutt.

I'm not aware of a sync service but it would be pretty straightforward to
build. I'll have a look at it sometime, probably taking code from the FastMail
app for Android, which has a partial JMAP client in it already.

I need to pull this info together into a "software" page on jmap.io - I'll try
to do that soon. In the meantime, if you're working on something that uses
JMAP, let me know :)

~~~
robn_fastmail
Done: [http://jmap.io/software.html](http://jmap.io/software.html)

