
Ten Commandments of How to Write an IMAP Client (2006) - macintux
https://www.washington.edu/imap/documentation/commndmt.txt.html
======
Felz
IMAP was an okay protocol for its time, but in my experience it's basically
mindbogglingly bad by current standards. The root of the reason why is the
whole concept of "message sequence numbers", which as far as I know are
extremely difficult to implement in a scalable and performant way.

Basically, all messages in a mailbox are numbered from 1 to N, and you can
reference a message by number, and when a message is deleted all message
numbers above it shift down by one. But since this could confuse clients that
are sending inflight requests, you can't start translating from the new
shifted-down-by-one numbers until you've told your clients about it, which
could take a theoretically unbounded amount of time.

One implementation of this I've seen is to get a list of every UID in a
mailbox at time of selection, and then update that as clients are informed of
new or deleted mail. Needless to say, that's a bit problematic with short-
lived or multiple connections.

So if your mailbox gets too big, too bad. This ends up complicating the
client: probably influencing rules 1, 3, part of 5, and 8 listed here. (1) You
can't have concurrent mailbox access because it might mess up MSNs reified to
disk, (3) you can't have connections live too long because they passively eat
up RAM with queued updates and a stored MSN list, (5) there are perfectly good
UIDs to reference messages with that you have to learn alongside message
sequence numbers, and (8) because opening a new session can be heavy.

~~~
pjc50
It was bad even 20 years ago, I remember a discussion when we were trying to
implement something coining the phrase "University of Washington brain
damage". It seems to have been extremely tightly tied to the UW-IMAP
implementations.

The need for a better protocol is why JMAP exists: [https://jmap.io/spec-
core.html](https://jmap.io/spec-core.html)

Although I see their point about non-REST, I think there's still a case for
having a REST flavour available even if it may not be the most efficient.

I wonder if anyone's tried mail delivery over git repo yet.

~~~
hyperrail
> _University of Washington brain damage_

This phrase made me laugh, because UW imapd was largely written by Mark
Crispin, who invented the IMAP protocol!

It seems that in about 2007, Crispin forked UW IMAP to create his own IMAP
server, "Panda IMAP" [1], which at least one website claims to be "fully
compliant" with the IMAP RFCs while UW IMAP is not. [2]

[1] [https://github.com/jonabbey/panda-
imap](https://github.com/jonabbey/panda-imap)

[2]
[https://imapwiki.org/ImapTest/ServerStatus](https://imapwiki.org/ImapTest/ServerStatus)

------
yrro
Does #1 mean that there are IMAP servers where my phone and my desktop and my
laptop (etc.) can't all read my mail simultaneously?

#7 makes me sad. Evolution supports a 'virtual trash folder' perfectly, but
since I also delete mails in other clients, my mailstore is polluted by a
malingering selection of folders with names like Trash, Wastebasket, Deleted
Items and so on. The same remark applies to a 'virtual junk folder' and
malingering Spam, Junk folders.

The 'All About Calendars' file is a wealth of interesting tidbits about time:
[https://www.washington.edu/imap/documentation/calendar.txt.h...](https://www.washington.edu/imap/documentation/calendar.txt.html)

~~~
ts4z
> Does #1 mean that there are IMAP servers where my phone and my desktop and
> my laptop (etc.) can't all read my mail simultaneously?

[edited]

I believe that's the case. The ancient and venerable UW implementation of the
ancient, venerable, and awful Unix "mbox" format from the UW server has a lot
of these sorts of problems.

The UW server did not have this problem with some of the other formats. It
suffered a lot from supporting a lot of different formats in borderline
hostile conditions (i.e., users were assumed to have filesystem access to the
server so no improvement to locking or caching was practical).

None of the current commercial implementations suffer from this, but I suppose
I can say that the lowest common denominator is still pretty low.

I may be overstating this because what I'm describing would also outlaw
message delivery, which I don't think is the case. I think one of the formats
did have problems with trivial concurrent access but I was never a UW-imapd
expert. I am sure that the mbox format has problems with EXPUNGE (removing
messages flagged deleted) as well as many other issues.

I guess it's also worth reiterating that STATUSing a SELECTed mailbox (one
that's open in another connection) is a pretty big waste of server resources,
as someone else noted.

------
billpg

        "Hello IMAP server. I'd like to deal with the message with this UID."
        "The UIDVALIDITY has changed since then."
        "Can you tell me the new UID from my old UID perhaps?"
        "No."
    

(Okay, I last worked with IMAP back in the 90s. Has modern IMAP since added a
persistent ID that will stay with a message?)

~~~
throw0101a
OBJECTID extension:

* [https://tools.ietf.org/html/rfc8474](https://tools.ietf.org/html/rfc8474)

Though if your mail server loses the UID-to-message mapping index and has to
rev the UIDVALIDITY, that's a more a problem with the implementation IMHO.

~~~
billpg
I'm an IMAP client and my user has clicked the delete-message button.
Accordingly, I've sent a command to make the change and I'm waiting for the
response from the server.

Oh no! Before the response came back, my connection to the server died. I
don't know if my request arrived and was accepted.

Time passed and I've just now managed to reconnect. I need to know if the
delete happened or not.

~~~
throw0101a
Check to see if the \Deleted flag was added to the message.

~~~
billpg
Which message? I have a UID that's no longer valid and the message IDs have
all moved.

------
robin_reala
What’s the current status of the JMAP spec? Are there any available clients or
servers?

~~~
earthboundkid
FastMail uses it internally, but I don't know if they've gotten any external
adoption yet.

------
codr7
I've dealt a lot with email along the way, trying to use it as an async
transport layer for higher level messaging etc.

At one point I had wrappers around Python's IMAP/SMTP implementations that I
excuted from Go, since at least Python had 20 years to make theirs work
properly.

Any other implementation I tried failed somewhere along the way, especially as
soon as you start moving messages around in folders or anything else beyond
simply fetching messages.

SMTP is simpler from my experience, but then so is the work it performs.

I even played around with the idea of cooking my own to get out of this loop,
but reading the specifications soon killed that idea. It would take a life
time, probably be deprecated by the time, and still wouldn't work properly.

The new JSON-stuff honestly doesn't look much better from what I've seen. It
seems to still be doing IMAP, only using JSON instead, which wasn't the
problem. I'm not holding my breath about it becoming standard enough to depend
on.

It's a mess, so much that I'm thinking it would probably be better to start
over entirely.

~~~
afiori
> The new JSON-stuff honestly doesn't look much better from what I've seen. It
> seems to still be doing IMAP, only using JSON instead,

regarding JMAP that should not be the case. From their FAQ the intent is to
make it easier to use already common workflows.

~~~
codr7
I do hope you're right, convenient protocol access to email would change the
game.

------
dqv
This is kind of related...

So I'm working on an app that uses IMAP. I need to know which emails I've
already processed. Has anyone ever used the message ID header to identify an
email as unique/seen? Or do you hash the email in its entirety? Or is there
some other way to do it?

I guess what I'm still confused about is determining how to handle a UID
validity change. So I'm afraid to use UIDs.

~~~
skshetry
How about adding a flag to the emails that are processed? It's fairly well
supported.

I needed this when i was working on my engineering project, and added
'processed' flags for already processed items. This way i could query all
unprocessed items with "UNSEEN UNKEYWORD <flag>".

~~~
dqv
Nice! I'll take a look. Thanks for the advice.

~~~
ts4z
This is a good plan, probably a notch better than using UIDs. You can use UIDs
as a cursor to the server as a hint to speed up the operation if that’s what
you meant in the other thread, and that’s a great thing to do.

------
mjevans
I don't know how many of these Outlook (any version ever that I've touched)
violates, but I do know that it AT LEAST violates #9 by default.

It irks me to no end, and un-checking that accursed options box in the IMAP
folders dialog is something no user should ever need to do to find their
"hidden folders".

~~~
taneq
Obviously there's some significant benefits to Outlook from an IT
administration side (or it turns into a completely different beast when used
with Exchange instead of IMAP) because as far as I can tell, it's otherwise
awful on any number of axes.

~~~
a2tech
Outlook is a good email client only when used against Exchange. It should
never be used as a stand alone client because while it CAN work as a POP3 and
IMAP client, its terrible at it.

Its not even a good email client when used against Exchange but its 'good
enough'

------
moonbug
That is unreadable.

