Hacker News new | past | comments | ask | show | jobs | submit login
JMAP – a better way to email (fastmail.com)
410 points by brongondwana on Dec 22, 2014 | hide | past | web | favorite | 115 comments



Great job guys! Things like this together with CalDAV and CardDAV are reasons why I pay you guys for service.

I just tried the following experiments with my Gmail and FastMail account. There's a reason why FastMail just feels faster!

  Try this:
  1) While having the Gmail iOS app open, mark an email as read on the web
  2) See how long it takes for Gmail to reflect the change
  3) Now try with FastMail iOS app, see that it's faster

  Try this too:
  1) While having the Gmail iOS app open, delete an email on the web
  2) See that Gmail app never updates until you manually refresh
  3) Now try with FastMail iOS app, see that it happens instantly

  How about the other direction?
  1) Read an email on your phone
  2) See that Gmail on the web never updates until manual refresh
  3) Now try with FastMail... Instant update!


Thank you for this comment. I do wonder why they simply didn't say this at the beginning of the article. I understood what they were up to purely because I'm a technical guy, but still I didn't want to actually have to watch the video -- namely because I'm deaf, and far too often there's no captions to explain what's going on so I prefer to read (and so do people at work, school, no sound etc), and it takes a lot less time!

I'm really considering moving from Google Apps to Fastmail... who else has made this jump? Totally worth it?


Their email sync is definitely superior to Google Apps, and now that they're adding DAV support the only thing really holding me back from switching my personal domain email over to Fastmail is email tagging. Gmail is, as far as I can tell, the only show in town for having email tags that sync across the server and client (unless you're in Exchange, but that's suboptimal for my personal use), and that's how I do pseudo-GTD to keep track of actions in my email.

Edit: I have a secondary email account through Fastmail where I don't need the tagging ability, and have been very pleased with the service. If you don't need tagging, I think it's worth the switch. They're clearly active in improving the service, while Gmail feels like it's moribund.


True 2FA is holding me back. Alternative logins are great but I'd prefer to force 2FA on the account.


They've got "enhanced multi-factor authentication" on their roadmap for 2015. Not sure if that's "true" 2FA, but it's on the way, FYI.


Every time I read about Fastmail, I want to make the jump from Google Apps. However, I don't do it because of Google Drive/Docs/Spreadsheet. Any recommendation for their replacement?!


I know it is not the solution, but maybe it will work for some people. You can sign up for Google account with external e-mail. It gives you everything but mail.


I know what you mean. Fastmail sync is definitely superior to Gmail for me, and DAV support may push me over the edge to switch my domain to FM. For Google Docs etc., maybe a private cloud provider that lets you add apps--ownCloud or something like that? I'm not sure if any of them provide apps at the same level of maturity as Google's offerings, but they could work depending on your use cases.


Yeah. I backed sandstorm.io and been closely monitoring it. The day collaboration apps on it closes the usability gap, I will switch.


Ha. I'm the same, every time I read about them I want to make the jump - but I'm not reliant on any of those, except mail fortunately.

I just haven't quite done it yet. From people I've spoken to, everyone praises Fastmail... seems like we all need that extra push.


Don't forget there's a 60 day free trial. How's that for a push :)


Less than pleasant first experience. Card failed, reason never communicated clearly enough. Very slow and archaic support.


I've had bugs reported, confirmed, and patched in production in less than 48hrs multiple times.

Slow?


Archaic?


Fax documents and payment information, poorly formatted support replies, supervisor escalation system after a couple different people over 2 weeks couldn't address a simple on-boarding issue.


It would be cool to have a browser plugin that decorated youtube links with captions.


The Youtube page has the full script on it.


Thanks for the heads up. I'm not sure about others but I almost -never- click through to the YouTube page unless I was seeking for more information but in this case, I just assumed that all the info was in the article (more or less).


This is good, and I love FastMail, but the one thing that really, really gladdens my heart with this is that they're pushing it as an open standard.

Thanks FastMail. Sincerely, thanks.


Amazing.

Modern, innovative, service-oriented products and a business model based on customers giving the company money in return for services.

I hope that this is a reminder to startups out there that you don't have to trick customers into handing over their personal information all the while pretending to offer your services "for free", in order to gain market share and be profitable.

The "traditional" business model can be quite successful.


I hope you're right but I fear that Fastmail isn't successful in a way that startup founders are going for. Advancing technology unfortunately isn't a priority for most founders.


We're fully owned by the staff, with no external funding. We've been nicely profitable since the beginning. Not huge profits but enough to pay for a nice office, decent salaries and few gadgets.

So yes, if your goal is to attract massive amounts of venture capital and then get bought out, don't copy us. If you want to have some fun doing something you love without having to worry about anything else then this is definitely the way to do it :)


Why would you create a new email protocol without an encryption requirement? I understand trying to fix the existing protocol problems but one of the biggest is plaintext.


Its expected that you'll run JMAP over HTTPS which takes care of the wire encryption. For end-to-end encryption, there's a heap of problems that still aren't solved (key distribution, need for content analysis). There's no reason JMAP couldn't be extended in the future to support these via additional properties, attached encrypted payloads, etc but mandating it from the outset would quickly see the whole protocol dead before it began.


>key distribution GPG key servers?


And how do you trust those? And if I'm a brand new user, how do I get my key in there?

All the moving parts probably exist, but so far no one has pulled it together into something that a random person off the street can use.

Its not the problem JMAP is trying to solve.


You don't need to trust the keyservers since you verify the fingerprints of downloaded keys against the fingerprints given to you in person.

The keyserver protocol includes commands to include keys in the keyserver network. How you send keys depends on the UI of the tool you use, geeks will do this:

gpg --keyserver <keyserver> --send-keys <fingerprint>


Right. I'm a little rusty on the details. My point stands - key distribution is not a solved problem for the average user.


It is solved for some specific platforms, GNOME for example:

https://mail.gnome.org/archives/gnome-announce-list/2014-Nov...

The average user doesn't use GNOME or Linux though :(


Because SSL does the job of securing data between the server and the client just fine.

There are a ton of interesting problems in the email world, including all the server to server communications. JMAP is lightning focused on solving ONE of those problems - the communication between end-user devices and their mail server - and doing it well.


You're assuming SSL is implemented without any security bugs. We saw what happened with Heartbleed and when SSL implementations are not properly reviewed and updated by a full funded team.


So you're assuming that an independent encryption layer that has to be reimplemented by every email client dev is going to be more secure than a widely studied protocol, implemented at the OS level. Right…


No, I'm not assuming anything and didn't even say anything like that.

My point is that using SSL on its own as the only line of defense should not be an excuse not to have anything else.

It's like saying my apartment doesn't have a spec for a security system nor a safe because the lock on the door does enough of a job to secure it.

Also, a widely studied protocol spec means nothing. The bugs are from the humans coding the implementations, it doesn't matter what level it is, they will have some bugs. Nobody can code a perfect secure implementation but we can have some kind of redundancies in the system, where if one security level fails, the rest can still have some reasonable security left.

Relying on SSL alone is not enough. But I don't think JMAP is the right place to do this, we may need something else in addition to JMAP.

At least JMAP is extendable, so that's one good thing it has.


If SSL is broken, you are - as many people have noticed, screwed. JMAP itself is entirely encryption layer agnostic. It's transport layer agnostic. JMAP over HTTPS is definitely going to be the first layer, but we're looking at websockets with interest as well.

If you were insane, you could do JMAP over XMPP, or JMAP over email. That would be neatly recursive...


In case you hadn't noticed, the authenticity model of SSL was an afterthought and is completely broken. I recommend listening to Moxie's talk about this:

https://www.youtube.com/watch?v=pDmj_xe7EIQ http://www.thoughtcrime.org/blog/ssl-and-the-future-of-authe...


Everyone has noticed. This cannot be solved overnight and rolled out to the entire planet. Stop beating this dead horse.


I don't think that's very relevant for this use case, though - JMAP is clearly intended for use by custom clients, not browsers, and those can use SSL with a completely different model from the CA scheme, including bundling certs for the most popular providers (similar to HSTS preload lists).


Exactly what I was trying to say but likely did it wrong.

I rather have a secure modular system than a single protocol that does everything. We simply just can't leave it as JMAP+SSL because once SSL cannot be trusted, it should be easier to replace it with a better solution or an updated SSL protocol. I don't think it is reasonable to include this within JMAP and force all services to update jMAP overnight because the internal crypto library had a hole.


What would you recommend then?


No clue at the moment, this is still a big problem and something Fastmail already knows based on the comment here from robn_fastmail.

If we have a solution, everybody would be using it by now. PGP or SIME was going to be it back in 90s but it didn't take off well.

The content of the message itself should be encrypted at the very least but how do you deal with the keys then?


That presents an interesting problem: can any implementation of a communication protocol enforce a requirement that arbitrary payloads are cyphertext?


This is for the client to speak to the mailstore, where plaintext is often not a problem. In many cases, it's a feature: if I forget my password, I'd like to be able to recover access to my email; at a company, I'd like authorized sysadmins to be able to get to other people's email, etc.

If you were designing a replacement for server-to-server SMTP, it'd be nice to require encryption, possibly end-to-end encryption.... although encryption can also be layered on top, at the RFC 822 level instead of the RFC 821 one (as S/MIME, PGP, etc. do). But this isn't an SMTP replacement, it's a replacement for IMAP (and ActiveSync, POP3, etc.).


For the well acquainted -- how does this relate to Inbox (https://www.inboxapp.com/)? I have not spent enough time with either to make a comparison


I haven't actually used either of these products, so I might be totally wrong, but....

Inbox is a separate layer on top of existing email servers. My impression is that the pain point they're addressing is, you want to build some awesome email integration app and sell it to companies, but you're looking at how complicated it would be to speak IMAP and IMAP-with-quirks and ActiveSync and the Gmail API and everything else (in order to build a high-quality integration) and that's scaring you. You can outsource that problem to Inbox, they'll make sure they get good-quality data from any email server reliably, and you just speak an HTTP API. This involves deploying a separate server which caches email. You talk to the "sync engine" over a reasonable protocol, and the sync engine takes care of talking an unreasonable protocol to wherever the data actually lives.

JMAP is an API to directly access an email store. It's something that an email provider (Fastmail, most obviously) will directly support, but it also only helps you for those providers that actually speak JMAP.

If you're an email server developer, an ISP, etc., then JMAP is cool and Inbox is not really useful -- there's not much of a point in having an email server that speaks IMAP and then also sticking a frontend to convert IMAP to something else. You might as well just make your email server speak a reasonable protocol directly.

If you're an app author, Inbox is useful today, and JMAP is only useful, maybe, if all your users are Fastmail users.

If you're an in-house developer at a large company that's running Exchange, and realistically you have no hope of getting the email admins to switch email servers (and you wouldn't want to run the email service yourself, even if they offered), JMAP is useless to you, and Inbox is super compelling.

(There is probably an argument that the Inbox "sync engine" should speak JMAP on the front end / those APIs should converge. Certainly the Inbox sync engine should speak JMAP on the back end, so Inbox can advertise support for Fastmail.)


Michael from Inbox here— a few key differences, from the horse’s mouth. :)

First of all— I want to say that we’re huge fans of Fastmail and really admire their team and focus. Rob and Neil actually came by for lunch this summer and hacked in our office for the day. We’ve discussed JMAP back and forth with them for a while, and specifically had a long thread when we added calendar endpoints to the Inbox platform. [1]

The fundamental differences between the Inbox APIs and JMAP are the kinds of applications built with them, the open source code, and the email providers supported.

The Inbox APIs are really designed to let developers access email data in non-traditional ways. Our site reads “Say goodbye to IMAP” but we’re not so much replacing IMAP, as creating a new protocol which liberates email data to a higher form. We think about email as “the database of your life” which means unconventional “clients” and access/manipulation patterns.

That’s one of the reasons we have a more REST-inspired design, whereas JMAP is closer to RPC. The Inbox API endpoints strive to represent canonical objects that developers simply treat as resources. JMAP is tailored specifically for traditional email clients, so it excels at fast operations (like archive -> reload), but compromises in other areas of simplicity and clarity IMHO.

The other main difference is code. The Inbox sync engine is open source with about 15k lines of code on GitHub.[2] You can run it yourself right now, and it works on all providers, including Fastmail! That means your users don’t need to be on any specific provider or backend. If you don’t want to run it yourself, you can use our hosted version which also includes support for enterprise environments that use Microsoft Exchange. [3]

I’ve been lurking on the IETF mailing lists for several years now, and one thing I learned is that spec documents hardly move the world forward. Developers need code that is battle tested and they can depend on. What use is a API spec if you can’t build an app with it?

The JMAP system is a great spec for FastMail’s internal mail service, and it works to power their own clients, but unless they make some drastic changes to the code and business, it’s not a platform for building 3rd party mail apps.

As an aside, I totally wish I had more time to write. I envy how prolific the Fastmail blog is these days. :) Keep up the great work Bron, Rob, Neil, and the rest!

[0] https://www.inboxapp.com

[1] https://www.inboxapp.com/docs/api#calendar

[2] https://github.com/inboxapp/inbox

[3] https://www.inboxapp.com/features


Hi Michael,

Thanks for commenting! I agree that our APIs are orthagonal in some ways.

We definitely plan to ship open-source code for everything JMAP. I'm working on an IMAP<=>JMAP proxy, which I'm hoping to have in time for FOSDEM in February. I'll be giving a lightning talk about JMAP there.

I tell you what - that writing it taking a lot of my time! Evenings, on the train to and from work. It's not a pace we can keep up forever.

You guys are doing awesome work. Hope to catch up with you again - and hey, it will be great if you add a JMAP backend when JMAP is a bit more mature.

Worst case for FastMail, we still have an awesome API for our own backend. Best case, we help everyone else too.


This looks interesting, as long as it is more efficient as a protocol and API. On another note, the entire email infrastructure need to be redesigned in my opinion.


> as long as it is more efficient as a protocol and API

It is :)


I hate to be critical considering the considerable work they've put in, but I don't buy their "JMAP is actually more REST-like than most “RESTful” APIs" statement. But maybe I'm missing a whole lot.

REST doesn't preclude doing multiple things in one round trip. Well it does seem JMAP is a bit more flexible when it comes to multiplexing very different actions. But the error handling doesn't seem to tell you which command has failed. You still can't get away from certain ordering issues. And to be honest, I don't see how it's going to be that easy to cache.

This really is RPC – using it as a straight replacement for IMAP might work well, but I wouldn't want to use it as a platform to build something very different than an email client on it.


Each call is tagged by the client and the same tag is added to the response by the server, so you can tell exactly what error corresponds to which call.

Caching is pretty easy: you just keep a cache of each object type. The delta update mechanism means you can very accurately invalidate what you need to keep it up to date. The system is actually really flexible and a great fit for any CRUD based app; JMAP is really just a combination of a very powerful and efficient database access protocol, and the definition of some objects to represent email, contacts and calendars.


Not just some objects, but different methods to access different objects, and it's those methods that implement the semantics - the base of JMAP is just an encoding for bulk RPC.

What I mean is, if tomorrow I add a file with the mediatype application/vnd.icebraining to my server, an intermediate HTTP proxy can still cache it even though it knows nothing about my format.

But if I add a new object (say, Note) to my JMAP server, along with the appropriate methods (getNotes, getNoteUpdates, etc), an intermediate JMAP server will be unable to cache them, since it needs to know the semantics of the new methods.

So frankly, I don't see how their protocol follows an uniform interface at all.

I still think it's pretty neat, though, don't take me wrong. I just want people to stop abusing REST. There's nothing wrong with RPC!


While I agree with you that in general there's nothing wrong with RPC, I'm not sure what the advantage in this case is. You seem to have an idea, though…?


Oh, you'd have to ask the JMAP devs. But I do think they'd have to bend the constraints pretty hard to have a RESTful interface while keeping the same advantages w.r.t. efficiency in the number of requests and data transferred.

Like Fielding writes in his dissertation, "The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction."


Oh right, now I see the: ["error", { type: "unknownMethod"}, "client-id"] -bit. Still, it sounds like you're reinventing HTTP verbs and statuscodes and embedding them inline in the JSON just to be able to multiplex the calls. Might as well use HTTP2 or SPDY, then.

Also, caching JMAP may be easy on the client side, but not for any intermediate cache - unless it understands JMAP.

I'm not saying RPC over HTTP is bad, mind, but you do seem to lose some of the benefits of HTTP.


Approximately nobody likes having intermediate caches that can cache their https traffic.

Approximately even fewer of the people here at HN like the idea of their email messages behing sent over non-secure channels.

At least we're not embedding HTTP status codes in XML...


I haven't had a chance to review the specifics of the protocol yet, but as a Fastmail user, I can vouch for the speed of event propagation across multiple clients. Its lightning fast; noticeably faster than say, Gmail.


Wasn't there a startup recently that pivoted from being a mail client into building a generalized API for interfacing with mail services like gmail/yahoo/outlook? I forgot the name but I'm super interested in efforts like this.

I have the feeling the next big social network disruption is going to be leveraging e-mail in a big way. E-mail is the established quasi-p2p (it's a network of centralized services) platform that everyone has an account and a very complete social network on. I'm not saying it would be easy to launch a Facebook competitor from e-mail, but I believe it's still got the potential to launch one last social network and someone just has to do it.

I'm hoping the key lies somewhere in building the right sort of mail client, that's got the reliability and extent of e-mail, adding on top some hook feature and layering in perhaps a dash of privacy, cryptography and independence. There's got to be something viral in there.



We didn't pivot. ;)


IMHO the big outstanding problems with email are spam and the difficulty of using public-key cryptography (both key distribution and mail client support). Write a dead-sexy mail client that makes the crypto completely invisible and uses either a public key directory or some kind of OAuth-ish thing, and you'd solve the latter. Better, if you made finding someone's public key take time and/or money, and simply refused to deliver unsigned mail, you'd take a big step toward killing spam.


If only they could make encrypted emails as transparent as encrypted SMS like Textsecure ...


Did something happen to the comments? I'm seeing 78 upvotes and no comments.


I felt a great disturbance in the Force, as if millions of people raised their hands in joy and suddenly clicked to upvote. I fear something incredible has happened.

Better and more robust email protocols, based on unencumbered open tech? Yes please. What else is there to say? :)


Fastmail uses Hacker News to spam, I mean publish all their new developments.


In general I dislike FastMail promotion on HN. But this article is different - it's some new open development, potentially very useful to many users, and its success is directly related to the number of developers who will pick it up. So at least this time it deserves some attention.


This looks fantastic and is a huge, long-overdue breath of fresh air in the protocol space. But realistically, what will it take for major vendors like Apple and Google to start adding support for it?


Neat. Two questions about the digest:

1. You recommend using a digest of the message to generate the message ID. Did you consider mandating that scheme or going farther and using a Merkle tree for the mailbox representation? It seems like this would allow for generating single requests that can fetch all new items.

2. Why SHA-1?

Edit: Also, thanks for keeping Fastmail running. As a customer of about ten years it's much appreciated.


No mandated format for message-ID. If you are proxying an existing store with a stable message-ID, you can just use that. My partially written gmail proxy uses the MSGID from Gmail, for example.

SHA1 was a good balance between speed and unforgability at the time. There's still no known collision, and I'm working on putting a piece of randomness into the LMTP headers that are added to every message on delivery into Cyrus to make it even safer.


I understand you guys use and contribute to Cyrus. Is there a JMAP interface planned for Cyrus? Or for any other mail servers, like Dovecot?


We're working on a proxy that will work with any IMAP server that of course will work with Cyrus. I think its likely that we'll look into building support directly into Cyrus at some point but we don't have any concrete plans yet.


http://git.cyrusimap.org/cyrus-imapd/tree/imap/http_jmap.c?h...

Totally a stub right now, but yes - we would like to have it directly in Cyrus...


from the article itself:

"Finally, we know IMAP, SMTP and the DAVs aren’t going away any time soon. No protocol will succeed unless it provides an upgrade path from where we are now, and a compelling reason to switch. We will provide a proxy which can talk to existing servers and present them over JMAP."


Your comment implies that the quote you provided answered my question. It didn't.


It's missing the one killer feature that would fix the spam problem: approved senders. If you had a "friends" list like Facebook where only people you've approved may send messages to you, we'd cut down on 90% of spam immediately. Between that and the lack of mandatory encryption this is just another tedious protocol to implement.


There's heaps of problems to solve in the email world for sure. JMAP isn't trying to solve all of them. It's trying to make one very well defined part - the communication between end-user device and server - a whole lot better.


That's not a practical feature in the real world. Sometimes you need email from unapproved senders. Spam filtering is better accomplished before or after delivery, not during.


There's nothing preventing an unapproved sender from requesting contact privileges, but once denied, their messages and further requests would be ignored by default.

Each server and user would have their own identity management concerns.

Basically someone can send a "friend request" and if approved by a human (the recipient) they may communicate freely from that point.


How is this different from adding an address to your blacklist after the first message?


They can't deliver their spam message to begin with. You'd need to approve the contact then you could still block them if they spam you.

There's still friend-request spam on facebook, but not nearly as much as email because you can just ignore the people you don't know and block people you do know if they begin to spam you.


The request for contact would be just as bad as the spam itself...


How do you deal with validating a sender? That is, how do you ensure that spammer's don't just create tons of fake new senders?


That's a great topic to have a conversation around.

My initial thoughts are that you'd have a blacklist for servers as well as individuals. So if people mistakenly approve senders at around the same rate they mistakenly open spam emails then you'd only get through one time in ten thousand. If your local messaging provider realizes that 9999 out of ten thousand friend requests from a particular domain are denied, then they'd block the entire domain.

Since a friend request conveys little to no information beyond the fact that someone wants to communicate, they'd be unable to get their message out effectively. If one in ten thousand spam friend requests gets through by mistake and after the initial spam message one in ten thousand isn't immediately blocked, then the spammers would need to set up ten thousand domains, each sending out ten thousand messages to get through to one schmuck who might fall for it, and then they'd need 9999 new domains since all but one have been blacklisted. It would likely be enough of a deterrent to prevent most spammers from continuing, although there will always be some amount of spam.

At least I don't think most people would be likely to accept a friend request from IncreaseYourDickSize6969@nigeriamail.com -- but there may be more effective strategies for handling burner identities. How would you do it?


try sanebox, this helps a lot.


> By using the platform push channels, JMAP avoids having to hold its own connection open.

Can anyone explain what they mean by "platform push channels"? I don't understand how you can have realtime updates without leaving a connection open or periodically polling. Do they mean that they use the same mode of transport as Push Notifications?


Android, iOS, (Windows Phone, Blackberry, etc) have services where a server application can push a small data packet to a specific device. That device wakes an application in response which can then do things.

We use this right now in our mobile apps. The apps respond by calling back to FastMail to check for updates.

More info:

http://blog.fastmail.com/2014/12/02/dec-3-push-it-real-good/

https://developer.android.com/google/gcm/index.html

https://developer.apple.com/library/ios/documentation/Networ...


I' m interested in this from a security perspective. What does this new protocol offer in terms of better control around what makes it to the inbox? Would IMAP>JMAP translation before hitting the user give us better ability to filter out malicious items/spam?


I think the only thing this particular protocol should be concerned with is reliably providing a method for communicating between client and server. Most spam and malicious mail filtering should happen as early in the process as possible, hopefully during the initial connection.

There is a a protocol named Sieve ( http://www.ietf.org/rfc/rfc5228.txt ) which provides for delivery stage filtering rules. It is similar in capability and usage to procmail, but formalized and somewhat more modern (and not capable of running arbitrary system commands), and supported by some modern IMAP/POP servers, like Dovecot. Presumably if JMAP gets integrated into those servers, Sieve would also be available.

And, I would hope there wouldn't be an IMAP/JMAP translation layer, but instead servers would implement JMAP directly. Though I guess in the short term there might be some sort of proxy.


Sieve is the native filtering engine used by FastMail, so I'd say it integrates quite well with JMAP, from my experience. :)


So am I. From looking at http://jmap.io/spec.html#authentication, it looks like the password will be transmitted in plain text. (See the text below the 200 response)

That makes me extremely queasy. Yes, HTTPS theoretically provides transport layer security, but a single breach of transport layer security results in the attackers permanent access to your mail.

I.e. run a MITM attack in a coffee shop, snoop up JMAP passwords, make use of them at a latter point in time.

I really hope I'm missing something important, or misread the spec.


If you read the spec closely, you'll note that it provides support for arbitrary challenge/response auth mechanisms. Its conceptually the same as SASL in that respect.

Yes, we're assuming a secure transport. Most of the internet currently does. Most of the passwords you send over encrypted channels right now are plaintext. This is not something we're trying to solve with JMAP (if it even needs solving, which is debatable).


I wouldn't ask you to solve it. I would've hoped for simply deferring the problem to somebody else. (I.e. OAuth)

Authentication is an incredibly hard problem. Including it in the spec encumbers it. And allowing for arbitrary challenge/response auth mechanisms gains you exactly one thing - guaranteed incompatibility as the protocol gains popularity :)


Having worked in identity management and access control, I can't emphasize enough how important your second paragraph is, and why Fastmail's approach of leaving authentication out of the protocol is the right way to go. Over and over we run into solutions that are borked from the start because they have a basic, bare-bones authentication component that is included because customers expect to be able to log in "out of the box" but then can't be replaced with a sophisticated authentication module because that basic bare-bones piece has its crappy insecure tendrils entwined throughout the rest of the product. Mikhailt said it in this discussion, too--modular components that do their single job well, and don't cram them all into one protocol.


Plain text passwords. No amount of blablabla can excuse that.

Plain text passwords.


So all those services you use right now where you type in a password. How exactly are those passwords transmitted to the server?


The plaintext password could probably be replaced by hashed passwords. Directly within the spec. Even though it already allows additional or different layers of security, this motivates developers to implement a basic level of security right into the application layer.


hashed passwords are security theatre unless you have a challenge-response in which password is hashed along with a challenge from the server, otherwise you don't need the password - you only need the hash to gain access. I've MITMed a system like that before when I wrote a nice interface to the awful timesheet system at a previous job.

Mandating specific security mechanisms isn't future proof. Authentication is almost a side issue to JMAP itself. Getting a securely authenticated and protected channel is phase 1, sending and receiving the JMAP protocol items is phase 2.

The reality is that almost everyone is sending plaintext passwords over SSL these days, and the reason isn't that they hate you - the reason is that it means they can bcrypt the password on the server side.

An interesting factor about most of the challenge response protocols out there - the server side needs to know either the plaintext password or something pretty reversible about the plaintext password. That's how the server can have a little chat over the wire with your client about the shared secret that they both know.

Since my security protocol design credentials aren't that much greater than the average internet commentator, I don't trust myself to design, or the average client writer to implement, a fancy security protocol that nobody has used before. Tried and true please.


Mostly agree with you, but minor nit: The server doesn't need to know anything reversible about the password. A message digest is enough.

And your final paragraph is the reason I posted my original comment in the first place. Don't design a auth/security protocol when you design a mail protocol. Delegate to OAuth, move on. (Then again... debate rages there, too[1]. Auth is not fun(tm))

[1] http://hueniverse.com/2012/07/26/oauth-2-0-and-the-road-to-h...


What prevents an attacker from using the hashed password (without decoding it) in his/her own requests?


I guess this tries to solve the same issue as microsoft active sync, just in a open specification way.


FastMail continue to impress me. Good work folks.



My presentation to the Thunderbird folks led with that, and we would have used it at Inbox Love too if we'd had time, but a 5 minute presentation had to get straight to the point.

It's a good comic. Good point.

We didn't take this decision lightly. There are significant reasons why all the existing standards don't do what's needed, and we made sure we built a technology preview and used it in the real world first - check the video or try out FastMail yourself.


Funny, but how many open mail retrieval protocols are in common use? I can think of maybe two - IMAP and POP3, and a small selection of proprietary ones, dominated by Exchange/ActiveSync.

The time is right for a better standard.


What does this do that IMAP and SMTP don't do already?

All you need is an IMAP server, accessed from all your devices. The one that comes with Android isn't bad. Thunderbird works fine on the desktop.

You don't have to use GMail with Android; when you first power up your Android device, click "Later" when it asks for a Google login. Then delete the Google One-Time Startup app. Google won't bother you again.


IMAP is a terrible protocol. It's a poorly designed filesystem with all the baggage that comes with it. Read the IMAP overview in the IMAP ruby module -- it's disgusting and was never meant for the modern world.

http://rxr.whitequark.org/mri/source/lib/net/imap.rb


tldr; version: It reduces the number of network roundtrips (and server overhead) needed to keep clients in sync (great for mobile) and addresses the fragmentation issue between the various server & client implementations of the sprawling IMAP protocol.

By moving to JSON and HTTP it also makes the barrier to entry a lot lower for client developers and opens up the possibility to improve speed yet further with GZIP and websockets.

All that while being able to stay compatible with legacy IMAP servers via a reference proxy provided for free by Fastmail.

Edit: Oops I've just noticed your handle so apologies for assuming you didn't read the article. I guess you've read it and still don't see the benefit. In which case would love to know why you think it's a wasted effort (personally, I think it looks like a welcome refresh).


The Ruby version of the IMAP client isn't that bad. Yes, IMAP doesn't use the current fad for marshaling, JSON. But a few years ago the web crowd would have wanted XML, and a few years from now they'll probably want some binary protocol Google comes up with. "Gzip" compression isn't a big win for objects the size of email messages, anyway.

IMAP servers that don't use a real database are sure to be a mess, but that's not the fault of the protocol.

I'd rather not have some startup in the middle of the mail chain. They'll start thinking they own mail.


JMAP is a rather misleading name.

"J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but rather the use of HTTP, along with recently added HTTP features such as push. JSON is just the message packing scheme. HTTP is what does all the heavy lifting.

"M" is misleading because this protocol is designed for a lot more than mail. It also handles contacts and calendars, and I wouldn't be surprised if Fastmail made their file storage service accessible through JMAP as well.

What's even worse is that once you come up with a protocol that covers everything that Fastmail does, everyone and their dog will try to add more services to the protocol. Instant messaging? Got it. Collaborative editing a la Google Drive? No problem. And before you know it, the protocol is bloated as hell and you've basically reinvented HTTP.

Please don't try to do everything. Do one thing, do it well, and put a strict limit to the scope of the project for the time being. A few more minutes of battery life on a phone is not worth polluting the world with yet another example of massive scope creep. IMAP+SMTP using a stateless protocol would be cool, but anything more and I'm not sure.


> "J" is misleading because it's not JSON that distinguishes JMAP from IMAP, but rather the use of HTTP

Not actually true; the protocol does not require HTTP. Any bi-directional stream transport will work just as well.

> "M" is misleading because this protocol is designed for a lot more than mail.

Maybe "Messaging" would have been better. I don't know.

Part of the JMAP name is to make it look like IMAP so its easy to see the relationship. J is after I. And so on.

But lets not quibble about names. Perhaps its not a great name. It doesn't really matter because products get different, better names.

> reinvented HTTP > yet another bloated protocol

This is kind of "citation needed" stuff. I'm not saying JMAP is perfect, in concept or in implementation, but we've have a shipping product running on it for a couple of years. Its got runs on the board already, and we're inviting everyone to pick it apart and make it better.

> IMAP+SMTP using a stateless protocol would be cool

Both IMAP and SMTP are inherently stateful, and that's not the worst of their problems. Why not fix it all properly?


The name is somewhat misleading, but we spent ages and couldn't come up with anything that described it better, was snappy, and wasn't already taken.

By the way, I've spent years on the protocol mailing lists with people who claimed that making it possible to submit email via IMAP was creating a bloated protocol. Support considerations, pah - real world problems don't matter. Meanwhile everyone was moving to ActiveSync so that they could actually get shit done rather than congratulating themselves on the purity of their model.

We definitely do want a model that extends easily (the record update and state-based changes model) to everything that a client wants to do, so that a client can get push updates to the things it cares about in a single way. If this model extends past email, then yay for that. Interestingly, caldav with its sync-token is further along this path than most protocols - it just doesn't have an open push mechanism.

I strongly believe that edge-triggered updates are the right way to do push. Send a small packet with no confidential data through the push channel - it doesn't even need to be encrypted, and it doesn't matter if it comes through multiple times - the client can protect itself by only checking for updates every N seconds if there were any pushes during that time period. The only attack is to block the pushes, but that one can't be avoided by any system - if you can't communicate, you can't communicate.

The actual data updates are pulled through the same secure channel as regular communication. Again, if you can get a secure link, and communicate down it, JMAP works.

In our case we send multiple state strings at FastMail right now (one each for email,contacts,calendar) so the client knows which things to ask for updates on. In theory, just a single token would be enough.

HTTP push features are only used at FastMail in the desktop client. On Android it's using Google's push. On iPhone it's using Apple's push. It's trivial to add new push channels, because they're just an edge trigger.


First thing I thought: It's one better than IMAP and meant to replace it. I++ == J. JMAP.


You mean ++I == J


It must be a generational thing: for me, anything starting with "J" must have something to do with Java.

... But to be honest, I don't really care. A lot of people don't know what IMAP and POP and SMTP actually stand for -- they don't care, and neither do I.


I thought that it'd be lowercase "j" that signifies Java, not a capital "J"...


"And before you know it, the protocol is bloated as hell and you've basically reinvented HTTP."

I don't think this assertion makes sense. If it's JSON over HTTP (as you've said above), why would they need to reinvent HTTP to achieve any of the stuff they're trying to achieve? They already have all of HTTP to choose from, no need to reinvent it. And, you've made a lot of claims based on some kind of slippery slope argument (which is a fallacy).

I think it would be educational for everyone talking on this subject to spend some time learning a bit about the existing protocols in the space. SMTP is messy, and shows its age in a lot of regards (character encoding complexity, transfer of large data, authentication of sending servers, etc.). IMAP is somewhat less messy and slightly more modern, but still quite old.

Neither protocol is well-tuned for modern email usage. I don't know that JMAP will solve it, but given how much effort is going into web tools, both on the server and client side, and how many more people have experience developing with HTTP, I don't see any reason to not leverage that for email, too.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: