
How to Replace IMAP - mk
http://www.gaborcselle.com/blog/2010/02/how-to-replace-imap.html
======
ajross
Not to spend too much effort defending IMAP (which does, to be fair, kinda
suck) but... come on. It's not _that_ hard. It works just fine, enjoying
pervasive support with very high quality clients and servers available freely
on essentially all platforms.

A new protocol would be ... prettier, I guess. But to pretend that the reason
facebook and IM are replacing email is IMAP (and not, y'know, spam) is
borderline delusional.

~~~
sandGorgon
<http://jerakeen.org/notes/2010/01/yay-more-email-clients/>

_The Android GMail client is a perfect example of what a client looks like in
this world. It talks to the (secret / private) GMail API, it does offline mail
reading, and queues actions so you can archive / filter / whatever mails while
offline and it’ll push changes later. You can read and write mail. It doesn’t
try to do anything clever, because anything clever done on one client isn’t
reproduced on any other client. And if I don’t have a client on my current
computer for GMail, I can use a web browser, and still get all the features of
the server. I use the web gmail interface for everything anyway, because it’s
better than any GUI client I’ve got._

I dont know - this guy makes for pretty convincing arguments on why email
should be done right on the _server_ rather than the client. Which effectively
means a new protocol.

~~~
_delirium
If not designed very carefully, though, it can reduce the user's ability to
view things as they like them. With the current status quo of mailservers
_not_ handling message threading and such, I can choose a client that handles
it how I want. If the server is doing it, it had better either do it how I
want it, or provide some way to customize what it's doing.

(For example, I don't like gmail's flat "conversation" view; I much prefer a
"threaded conversation" view, like classic Usenet readers, or HN discussions.)

~~~
sandGorgon
reMAP claims to treat conversations as first class objects. Given suitable
APIs on top, I think you could build different views on different email
clients.

I think what is meant by a new protocol and server innovation is a change in
the data model. I do not believe they are going to dictate views.

------
JoachimSchipper
This is a collection of the author's favourite technologies/gripes, not a
useful proposal. A large number of errors can be found by considering that
it'll have to talk SMTP on the back end.

I'll only point to some "highlights".

OAuth isn't nearly secure enough ([http://hueniverse.com/2009/04/explaining-
the-oauth-session-f...](http://hueniverse.com/2009/04/explaining-the-oauth-
session-fixation-attack/)) for your e-mail (which, don't forget, gives access
to _everything_ via password resets). Let's just ignore the part about third-
party access.

Encoding everything as UTF-8 is nice, but breaks the first time some idiot
mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any
charset marker, obviously. (This goes right back to the "where does the mail
come from?" issue.)

IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and
parts of others), and all sensible servers support IDLE
(<http://en.wikipedia.org/wiki/IMAP_IDLE>). It also supports downloading MIME
parts separately, solving the part of #7 that can actually be solved (again,
you'll have to accept mail.)

~~~
gaborcselle
Thanks for your suggestions Joachim! This is why I wrote that blog entry in
the first place: To get some high-quality feedback.

I think IMAP could be replaced independently of SMTP. You could conceivable
replace an IMAP server with a reMAP server and not affect the rest of the mail
ecosystem. SMTP will likely not die for another hundred years.

1\. I know about the OAuth problems. Do you think we'll see a more secure
solution in the next few years?

2\. Is there an easy way to validate UTF-8ness? Happy to hear suggestions.

3\. I know about THREAD and SEARCH and IDLE but find them clunky. Under the
assumption that we're replacing IMAP with something RESTful you might as well
get those ones right.

Happy to take this offline as well - my email is on my HP

~~~
JoachimSchipper
First off, thanks for taking this well. It was a bit of a rant, even if it was
toned down from the first version. Sorry.

We agree SMTP will be around forever; but how do you plan on abolishing MIME
then? Your system will have to handle it, no matter what. Do you want to push
MIME handling to the server? IMAP already does that, there's nothing
revolutionary about that.

Due to the special structure, it's usually possible to distinguish UTF-8 from
other 8-bit text. The tricky part is what to do with unmarked non-UTF-8 8-bit
text - it could be pretty much anything. Mutt, for instance, allows the user
to configure this according to what the user expects to see. Quoting the
manual,

    
    
        For example, Japanese users might prefer this:
        set assumed_charset="iso-2022-jp:euc-jp:shift_jis:utf-8"
    

There is a lot more information about such things on the Mutt site. Basically,
Mutt tries to parse the headers/message according to a specific character set
and falls back to the next one if that fails.

I probably wouldn't trust my e-mail to a protocol that was only a few years
old. I don't anticipate any new security protocols with significant uptake
either; "serious" stuff already uses SSL/TLS plus an authentication method
(passwords/certificates/SASL), and the web crowd is unlikely to replace that
(OAuth is about weakening security in a controlled way, not replacing SSL.)

And yeah, parts of IMAP are clunky. But that's not enough reason to rip-and-
replace it.

------
nfnaaron
I can't say whether his implementation makes sense.

Regardless, I would like to see public key encryption of all email. There's no
reason why messages should be plain text anywhere except at the two ends.

I realize key management would be a problem, but it's probably a solvable
problem. The solution would have to allow most people to be practically
unaware of encryption issues.

~~~
webignition
Encryption of all email would make intermediary virus scanning impossible and
spam scanning next to impossible.

~~~
conover
Wouldn't it make spam impractical because you would all of a sudden need much
more processing power to encrypt the messages (especially millions of them)?

------
tdmackey
First the solution doesn't really fulfill all of imaps use cases (especially
push mail) but that is unimportant.

What is important is that the IMAP protocol isn't difficult, and I don't think
that is what limits "email innovation." It is more so that it has been around
since the 70s and email clients haven't been able to remove themselves from
their past. It isn't the protocol limiting them from moving forward, it is the
shear number of other clients that wouldn't support any of the "innovations"
one would try to bring to email.

~~~
beh
I'm very hopeful for Letters, a project just taking off under the leadership
of some pretty big names: Gruber, Rands, Brent Simmons. They're hoping to
collectively create a "lean and programmable IMAP email client, with plugin
and automation APIs, designed for developers and power users." While still
sparse and very early-stage, their vision document is ambitious:
<http://pastie.org/785269> If something comes from this, it'll definitely be a
step forward to modernize email clients.

Project wiki: <http://letters-wiki.heroku.com/>

------
viraptor
There are some good points, but:

1\. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies
in many places that will effectively block this protocol. Also custom TLS
connection gives some hope for domain-specific certificates instead of ip-
specific - there is enough problem with them in HTTPS as it is.

3\. People still don't know how to set the encoding (or what the encoding is)
and will happily send you Greek in latin1 or something equally silly.

4\. Would it really work for mailing lists where conversations naturally fork
into many branches?

5\. IMAP supports labels... even if it's crazy number-based labelling where
every client needs to set its own names for the labels. We'd just need to make
it not suck this time ;)

So it's not all that bad in IMAP - there are many areas where we can do worse
by accident. But yes - it would be nice to have server-side text searching...

~~~
pyre
> _4\. Would it really work for mailing lists where conversations naturally
> fork into many branches?_

I agree with this. The idea that messages _must_ be accessed by conversation
is a bit much. _Support_ for conversation group could be nice, but ignoring
that feature would have to be allowed so that clients could implement their
own views.

For example, the following structure would be crappy to force on clients:

    
    
      Label
      `-> conversation 1
          `-> msg0
          `-> msg1
      `-> conversation 2
          `-> msg0
          `-> msg1
    

Needing to iterate through every conversation to pull tease out the full list
of messages (in order to use a different conversation-grouping method) would
be ridiculous. Not to mention the question of whether or not conversations
should span labels and how that should work.

------
jwhitlark
My responses to your Design Outline.

1\. I'm not convinced that http/https is the best protocol. I agree that it's
the defacto standard, but if you're going to go through this, there might be a
better way. (Push, in particular, is a problem).

2\. Stateless is wonderful, agreed.

3\. I agree with JSON over XML, but what about binary data? That is the
weakest part of JSON, in my opinion.

4\. I like where you're going with this, but I'm not sure this is the exact
right way to handle it.

5\. Agreed.

6\. how about SHA-1 hashes of content? Done properly, that would eliminate any
duplicate messages, and could make it very easy to include other messages.
This might also allow content to be transfered between accounts only once,
dropping bandwidth and speeding delivery.

7\. Don't know enough about MIME to comment.

8\. According to your comments on this page, you're referring to long polling.
I agree that this use case needs to be addressed, but I'm not sure that this
is the way to do it. Perhaps multiple options here?

9\. Good idea, but do you think that stemming/language differences is going to
be a problem? What about other types of content? attachments?

The biggest thing missing from this list is some sort of Public Key
Encryption. Anyway, it looks like a good start.

~~~
pyre
> _3\. I agree with JSON over XML, but what about binary data? That is the
> weakest part of JSON, in my opinion._

What binary data? All attachments in emails are Base64 encoded. That's what
MIME is all about.

> _6\. how about SHA-1 hashes of content? Done properly, that would eliminate
> any duplicate messages, and could make it very easy to include other
> messages. This might also allow content to be transfered between accounts
> only once, dropping bandwidth and speeding delivery._

Developers should start to shy away from SHA-1 since it's already started to
get broken.

~~~
jwhitlark
>> 3\. ugh. Hadn't realized that. I like IMAP even less, now. (Told you I
didn't know much about MIME ;-)

>> 6\. I wasn't intending it as a security mechanism, but as a way to know
which bits you have and which you don't. Do you think SHA-1 is still a bad
idea in that case?

~~~
pyre
_I like IMAP even less, now. (Told you I didn't know much about MIME ;-)_

IMAP might do some sort of MIME-handling, but MIME is part of the email spec,
_not_ the IMAP spec. Hating IMAP because of MIME is like hating cars because
of the invention of the wheel (i.e. thinking that the wheel was invented as
part of the production of the initial automobile).

<http://en.wikipedia.org/wiki/MIME>:

    
    
      Multipurpose Internet Mail Extensions (MIME) is an 
      Internet standard that extends the format of e-mail
      to support:
        * Text in character sets other than ASCII
        * Non-text attachments
        * Message bodies with multiple parts
        * Header information in non-ASCII character sets
    

MIME is used to say things like 'this block of text is a base64-encoded file
attachment, but also to say 'this block of text is encoded with this charset.'

If you use Gmail, try the 'open original message' option on emails that have
attachments or multiple text parts on them (or even emails that have both a
text and an html version of the content).

Dealing with MIME is part of dealing with emails in general, regardless of if
you're interacting with an IMAP server, or composing an email to send to
another user on the same mainframe (i.e. just moving files around).

------
dlsspy
> All data that's ever sent to or received from the server would be in JSON
> format. JSON is much more human-readable than XML.

I don't need my protocols to be human readable. I need my computers to read
them and know that I've got good, bug-free, reusable, efficient, and easy-to
understand implementations.

This isn't an argument for XML over JSON -- I've made good use of both, but I
don't want to consider how easy it is to type up communication protocols when
I'm defining stuff the replacement for something I expect to take over the
world.

It's _by far_ easier to implement the memcached binary protocol as a client or
a server than it is to implement the text protocol. Easier to get right, more
regular, and just generally more pleasant.

~~~
regularfry
As far as I'm concerned, the argument for JSON over XML is that the parser
should be much lighter. No bloody entities to worry about, for a start.

------
Joe_Bananas
Ah, yes. A new email scheme that would require us to replace all clients and
servers in one fell swoop. I can see that taking off.

~~~
gridspy
All it requires is a major server (i.e gmail) providing it as an option and a
major client (say a iPhone email app) supporting that option.

~~~
there
oh, that's all?

it'll require an RFC or some other kind of standard, which will take a long
time to finalize. then someone will have to write a reference spec server and
client, then "real" versions in "real" languages, then get them both to the
stability, security, and scalability of current imap clients and servers. then
once big email providers (isps, google, etc.) start supporting it on the
server end (gotta factor in all those different operating systems,
authentication systems, storage systems, etc.), popular clients can support
it. then work out all of the minor implementation details that some servers or
clients get wrong (i think almost every smtp and pop3 server has hacks in it
to support the stupid things outlook express does wrong), and then once
everyone complains long enough, the iphone will finally get support for it
(while a small but vocal minority shouts that android already supports it).

there is a reason why smtp, imap, pop3, dns, http, and all of the other core
internet protocols are still around after 20 years and it's not because there
aren't any faults in any of them.

~~~
lief79
If you are going for a clean design, you can go in the opposite direction too.
I'm fairly sure Ethernet and html evolved from a working solution. Now,
granted there should be a really clean and open interface if we are going to
try and make it a standard, but it's often the easier approach.

------
Skeuomorph
First, I love reMail. Gabor breathes email and search. If you own an iPhone
and use Gmail or Google Apps, get reMail--you'll thank him.

That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when
we're discussing these, laymen may need to know a few of these seem over-
simplified.

> "TCP connections are great, but for transferring large amounts of email
> securely, HTTP is the way to go."

1\.
[http://wiki.answers.com/Q/What_is_the_difference_between_tcp...](http://wiki.answers.com/Q/What_is_the_difference_between_tcp_and_http)
\- HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the
proposed REMAP.

2\. I generally have at least 5 devices using IMAP (on Google Apps) at once,
and interleave my interactions with these devices arbitrarily, yet expect each
to show me the exact same state when I look at it. Keeping them all in the
apparently same state is the sort of intractable software problem bedeviled
with implementation details, and I agree throwing out IMAP and starting over
with lessons learned could be easier than, say, the RFC linked below.

3\. <http://ajaxian.com/archives/json-vs-xml-the-debate> \- Seems this point
is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown
in. Meanwhile, both XML and JSON are missing an important concept given
labels/folders and conversations: multiple references to single objects.

4\. Conversations or threads exist today as In-Reply-To: GUID and References:
GUID headers, among others, but see the JSON problem above. We don't realize
these headers are there because clients generally ignore these and try to
group on Subject instead. Gmail grouping respecting existing headers is much
better, but from usability point of view, Gmail's approach makes it
challenging to bulk delete individual matching messages across a broad set of
conversations. For example, to bulk move or mark all messages on a particular
date results in the full conversations being moved or marked. So I like this
idea as long as I can still optionally operate on sets of messages without
affecting the rest of the conversation.

5\. [http://googlesystem.blogspot.com/2007/08/organizing-chaos-
fo...](http://googlesystem.blogspot.com/2007/08/organizing-chaos-folders-
labels-search.html) \- It's not yet clear that people prefer tags to folders.
After all, monkeys don't expect a banana to be in two boxes at once. I don't
want to label anything with keywords. I want semantic search.

6\. Excellent point. Changing GUIDs out from under us is just lame. Having had
to resort to <http://github.com/rgrove/larch> recently to move a decade of
email onto Google Apps because Thunderbird was putting new IDs on each moved
message, I fully agree this problem is evil.

7\. Seems we're stuck with MIME until we get rid of every legacy email server
in the world, or control what servers our recipients use. Between the new
server and new clients, sure. But the server will have to know it to send it.
Granted, IMAP is for receiving, and SMTP is for sending, but something in the
chain has to know MIME.

8\. "Call" an HTTP endpoint that "returns" when messages arrive? Just
nomenclature, since IDLE is a "call" that "returns" when state changes. Either
way, the socket is held open, so neither of these is "push". Maybe Gabor meant
"posts back" instead of "returns"? In any case, under the hood, iPhone push
and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to
notify the phone it should poll (or in case of ActiveSync, maybe doing a "long
pull"). See <http://msexchangeteam.com/archive/2004/04/26/120520.aspx> versus
[http://tools.ietf.org/internet-drafts/draft-maes-
lemonade-p-...](http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-
imap-12.txt)

9\. Will the RFC specify a search results algorithm, or will the same search
return different results depending on the implementation of mail server the
client is connected to? We manipulate a "client" tool to perform the search
and access the email, so our natural mental model is of the client at hand,
not of the remote server. This lets the brain wrap itself around Gmail's web
search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all
returning different results. Practially speaking, search at the server does
make far more sense from both a data retention and security standpoint. I have
509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not.

Thanks for brainstorming this, and thanks for a kickass iPhone app.

~~~
Nwallins
_1.HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the
proposed REMAP._

Yes, of course. He is not claiming that TCP is bad, but rather that it is too
low-level. i.e. One wants an application protocol (e.g. HTTP, SMTP, IMAP)
rather than a transmission protocol (e.g. TCP).

The HTTP verbs should be sufficient for email, so why not leverage them?

 _It's not yet clear that people prefer tags to folders. After all, monkeys
don't expect a banana to be in two boxes at once._

Heh, dare I reply? Tags can be used as pigeonholes, and this can be enforced
if desired. But why? Tags are so much more useful because they are _not_
pigeonholes.

Here is a classic example:

Does my app config go in a system directory or an app directory or a config
directory? the executables? the daemon? the log files?

The classic approach is to put system-level config files under /etc, log files
under /var/log, executables and daemons in various locations, etc. Some newer
approaches (HomeBrew, GoboLinux, etc) keep everything organized per-
application.

Instead, keep every file in the global folder / namespace. Tag the executables
as exe, the log files as log, config as cfg, and also tag everything with the
app's name. Possibly the version, etc.

A file is uniquely identified by its tags and its name. Any combo can be used,
so long as it doesn't already exist.

You don't have to make painful decisions about the unique place in the tree
that a piece of information should exist. Just tag it appropriately and move
on. The tag system could even create an optimal concrete hierarchy based on
tag and tag association metrics if needed.

 _But the server will have to know it to send it. Granted, IMAP is for
receiving, and SMTP is for sending, but something in the chain has to know
MIME._

I'm not sure if this is GoF, but this sounds like the classic gateway pattern.
You put MIME gateways at the edges of the new email network. MIME-only servers
must go through the gateways.

~~~
eitally
Your intended point about pigeonholes can sometimes backfire within the
context of email. Consider a Gmail user who sends a query to a number of
different recipients, expecting unique replies from each. Gmail, because of
the like headers, will group all of the responses into the parent
conversation. Let's say the users tags the first response with a label
corresponding to that sender, and then the second response comes in. At this
point, the user is at an "oh shit" moment, finally comprehending the
implications of Google's labeling of conversations rather than individual
messages.

You need to be able to specify both parent and child tags to be able to
facilitate search and give users something that isn't too confusing re:
organization.

~~~
Nwallins
> _and then the second response comes in. At this point, the user is at an "oh
> shit" moment, finally comprehending the implications of Google's labeling_

I am having trouble understanding what the problem is, here. I also suspect
this may be a _Gmail_ problem rather than a _tagging_ problem.

~~~
AndrewDucker
It's tagging a conversation rather than tagging an email.

I frequently put different emails from the same conversation in different
folders, because they contain different types of information that I need to do
different things with.

GMail will not let you do that.

------
mrduncan
The description for #8 sounds more like long polling to me. For push email,
I'd envision something like registering a location to send notifications to
(web hooks).

~~~
gaborcselle
Yeah, it is long polling. I could be misinformed, but isn't the problem with
web hooks that you might not have ports open and could be behind a firewall?

~~~
mrduncan
Yea, I'd say long polling is probably a better solution in this case than web
hooks since clients (likely) could be behind firewalls, NAT, and be going on
and off line relatively often. Sorry, the title and description on that one
just threw me off a bit - I think you chose the best option though.

------
cturner
You could implement this as a service on top of couchdb. The oreilly couchdb
book that has just been released shows examples of developing apps using the
javascript engine that's inside couchdb. You'd get easy access to some
functionality by doing this - tagging, email versioning.

