

Help me kill IMAP and Exchange in a single blow - Navarr
http://tech.navarr.me/2011/08/so-i-came-up-with-this-really-great-idea.html

======
rjd
OK... I've been currently researching and prototyping an email based
communication tool myself. And what you have below is nothing interesting at
all. Infact it looks like you've just noted down features from current hype
and gmail and attached your name to it.

Its just using web tech push crap around. You don't offer an improvement of
the problem, just a way to implement the standard process using familiar
technologies. The problem you are solving is "I don't like IMAP" not actal
problems with IMAP.

Anyway the proposed stack for those interested:

\-------------------------------

\- RESTful API over HTTPS. (HTTPS should not be required but recommended.) -
Not sure how to go about HTTP Status codes. I'm using them below, but this
might be considered misuse. Must consult for verification.

\- Labels (instead of folders, however, tags can be hierarchal - like how
Outlook renders Gmail tags with a / in them)

\- JSON (Maybe allow JSON & XML, but require JSON)

\- UTF-8 (Only encoding type allowed - used for EVERYTHING, including API
endpoints and API variables USE UTF-8. Will create larger messages byte-wise
for european and asian languages, but definitely worth it) "Stateless"
(Whatever IMAP states are, we don't do this). We use Session Cookies, and more
precisely OAuth as the de-facto standard.

\- Conversations as Priority - Servers should attempt to indicate
conversations, and keep messages grouped together like a forum topic. This
MUST be done server-side.

\- NO MIME ENCODING. Mime-Types as headers are very important, but NO MIME
ENCODING under ANY CIRCUMSTANCES. Built-in Push - @gabor talks about clients
calling an HTTP endpoint that returns as soon as a new message/s arrive.

\- Full Text Search - SERVER SIDE

Addresses use [user]@[domain] syntax.

MSAP servers are looked up via an SRV DNS request for _msap._tcp.[domain]

Return is in format TTL [class] SRV [priority] [weight] [port] [target]
example: SRV 0 0 443 msap.koneko-chan.net SRV 1 0 80 msap.koneko-chan.net The
first is the HTTPS (SSL) endpoint, and the second is the HTTP endpoint. Lower
priority is prefered, with a fallback to higher priority available.

We should also recommend having an SMTP endpoint for the sake of graceful
degredation. Example of "working" endpoint: <http://msap-serv.koneko-
chan.net/source-code>

~~~
yid
Sounds a lot like gmail.

------
m0nastic
I was always partial to Dan Bernstein's Mail2000 idea:
<http://cr.yp.to/im2000.html>

But I don't know if anything ever came of it.

------
skrebbel
I like the idea, but it feels a lot like the W3C in the 90's all over again:
solving a perceived problem with a standard instead of a product.

~~~
Navarr
Well, I'm not in the position to make a product out of it, or I would. The
most I can do is program a server using the open source protocol in Node.JS or
PHP.

------
drdaeman
> All communication over HTTP / HTTPS

Why HTTP? And, for example, not BEEP? HTTP can't handle multiple concurrent
logical streams, thus can provide you with one resource per connection at a
time. Every separate HTTP (and HTTPS) connection requires a TCP (and TLS)
handshake, re-authentication and so on.

Also, long-polling is a hack, born because of HTTP limitations. Is there any
reason to do long-polling when you can just send a signal over bidirectional
connection?

This could be a unpopular opinion (lots of web developers out there), but I
believe, while it is perfectly possible to use HTTP, it is just plain ugly for
such job.

> Stateless

Is wrong for protocols where you require a state (authentication, folder/label
subscription and so on). How many stateless pub-sub protocols you know about?
Be it lightweight Redis Pub/Sub or XML-bloated AMQP and XMPP - they're all
stateful for a reason.

HTTP cookies do represent _state_. You're just taking a shortcut by re-
authentication with a session token. The state you keep could be only auth
data (so you'll end up re-sending, for example, what labels you're interested
in every time), but it's still a state.

------
klodolph
Each message is just a resource, and we have a fabulous protocol for
transferring representations of resource state -- HTTP. "Just add a push
mechanism et voila." Given how much better HTTP authentication is compared to
IMAP authentication, as well as easy support for multiple streams,
statelessness, etc... it's hard not to be sold on HTTP.

Of course it's not that easy. But let's not go inventing new protocols when we
can avoid it...

~~~
Navarr
If you'd read Gabor's original article as well, you'd know that this is more
of an API over HTTP then a TCP protocol. I guess I didn't make that clear
enough.

------
pornel
We don't have current mail protocols because nobody was able to come up with
anything better — we have them, because we need interoperability and it's
nearly impossible to migrate all servers and clients to a new protocol.

I think the best bet is to do to "IMAP5" in the same way HTML5 has been done:

• keep basic functionality backwards compatible with existing implementations
(GMail, Apple Mail, Outlook have to work or the protocol will get ignored)

• drop all useless/unpopular/insane features (can we kill UTF-7, please?)

• make de-facto standards and common extensions part of the standard (everyone
uses XLIST)

• add very clear and strict implementation guidelines (IMAP is particularly
bad at this and all "requirements" read like "Clients should do this, but
might do something completely different")

Going RSS→Atom route is going to end like this: <http://xkcd.com/927>

------
7952
I think the purpose of this protocol is to bundle other data into the smtp
requeat. Why not just make a standardised format that can be included as an
attachment? Or even add special tags within the text of the email? This would
achieve the same where you have client support, and not need to spend time
writing a protocol.

------
sthatipamala
It seems you're hoping for a community to emerge and help you. This is much
more likely to happen if you flesh out your idea and write up preliminary
documentation of your protocol.

A FOSS community is great when things are already rolling but it's up to you
to put in the effort to catalyze it.

------
icebraining

        Heliotrope is a personal email server. It provides all the functionality you
        want in a modern email client:
    
        - proper message threading
        - labels
        - fast, full-text search over all messages with a complete query language
        - support for signed and encrypted email
        - an extensible JSON-over-HTTP API
    
        Heliotrope is a backend service against which email clients / MUAs can be
        written. To use it, you must use a client. For an example client, see
        Turnsole: http://github.com/wmorgan/turnsole.
    
        WHY ANOTHER PROTOCOL? WHY NOT JUST USE IMAP?
    
        Because IMAP is terrible and you want all those features listed above.
    

<https://github.com/wmorgan/heliotrope>

~~~
Navarr
This is nice, but unfortunately the protocol itself is a second class citizen,
and isn't being structured as a synchronization protocol.

------
dlsspy
I came here expecting to see how to do this with CouchDB. It's already been
done a couple of times.

CouchDB provides data synchronization protocols and the simple rules for what
acceptable incoming data looks like. Properties within the documents define
all of your "labels" and any other features you might find interesting (which
are later trivially indexed for retrieval).

------
puredemo
Maybe you can call it, "Wave."

~~~
Navarr
Wave was a really neat idea, but it had no backward compatibility in place.

In theory, MSAP would address that and have systems in place for backwards
compatibility and communicate with the older SMTP protocol (and IMAP if
necessary) while things move forward.

~~~
jarofgreen
So why not start fleshing out some details on how a wave-to-email bridge would
work instead?

If someone else has already done some great work and made it open, brilliant!
Use it :-)

~~~
Navarr
Because Wave doesn't solve the issue I'm attempting to address.

Waves solves communication, obviously. And waves feel more like personal forum
threads than anything else.

But the true intentions of MSAP go beyond email and to synchronization. And in
retrospect I should have addressed that more in my post...

------
jstsch
Write a prototype client and server. Release. Follow up on feedback.

------
ojilles
Your first problem is marketing (no one to help you, etc)... So pick a name
for the project!

~~~
Navarr
MSAP! Nobody follows the links =p

------
hng
You should write an RFC ;)

------
nknight
> _I continue to push my creativity and find new things to throw in to the
> protocol, ideas that wouldn't have to be there when the first part hits, but
> that its extensibility would make possible._

This by itself makes me believe this is doomed.

Part of the problem with IMAP is that it's incredibly complex. That
complexity, and differing interpretations of that complexity, is a huge part
of what makes IMAP such a headache to deal with.

Simplicity is a virtue. Don't add everything you can think of, _strip out
everything you don't absolutely have to have_. It's fine if the protocol is
extensible, but your first and primary goal should be to have a system that
synchronizes email reliably and efficiently. Get that right, _then_ start
worrying about the extra stuff. Until then, you're just making life harder.

~~~
Navarr
You're absolutely right! Thank you.

I'll need to take extra care to focus on the main issue before extending it.

I just also want the extensibility to be a center focus, so that it can expand
to be a synchronization protocol instead of only an email protocol.

~~~
karl_nerd
Can someone shoot this thing down: What if we think 'data synchronisation
protocol' instead of 'e-mail protocol'? E-mail would then be an 'append only'
application of this. The protocols focus would be to register a piece of data,
keep track of the 'owner' and sharers and maintain synchronisation (preferably
in a peer-to-peer way). I know this problem is big, but once it's done in a
reliable way it would be simple to build a good e-mail-style communication on
top of it.

~~~
derefr
You mean... Google Wave?

~~~
Navarr
Google Wave wasn't a data-synchronization protocol.

------
kahawe
Before we go about replacing IMAP which only has two things going against it
(old, complex but not broken) how about fixing the SMTP shenanigans first
which would actually be more beneficial? SMTP is even older and arguably
actually broken and while there are a few usable work-arounds for things like
spam and faking senders etc it needs replacing way more than IMAP does. While
IMAP surely is not so "sexy" in the eyes of the modern geeks, it seems to work
just fine... at least good enough so that googlemail decided to offer IMAP
access. And while the same could be said about SMTP - it kinda works and I get
my mail - it feels like this small little thing that was turned into a mutant
spawn from hell with way too many things plugged in and onto it and it just
begs to finally be put out of its misery and be replaced with a modern
solution.

And Exchange and Outlook is one of my personal pet peeves: I have absolutely
no idea why people haven't refused using this bloated, locked in and confusing
p.o.s. already. I can edit and change attachments in an email I recieved,
there are 3 or 4 different key combinations for searching depending on where
in the interface I am and on the server side it doesn't get better.

~~~
AndrewDucker
The solution to SMTP is simple - an IMAP outbox. Anything placed in that
folder is automatically sent by the mail software (and then placed in your
Sent folder).

~~~
andor
That's just an alternative way of getting messages from the client to the mail
server. The problem with SMTP is that mail _servers_ use it to send messages
to each other, and that there is no authentication for servers. Since there is
no authentication, spammers can just connect to any mail server and dump their
messages. On the other hand, it's what makes email an open system, and enables
you to receive mails from any other server.

~~~
pornel
> Since there is no authentication

This is mostly solved. You can't just connect to any server, because your
"end-user" IP is likely to be in DUL/PBL or other policy-based blacklist.

Open Relay is now forbidden and SMTP from end-user to mail server requires
authentication.

For server<>server communication you need to use SPF at least. Although
authentication is theoretically optional, in practice you won't be able to
reach many servers without any.

