I use OfflineIMAP to fetch & emacs/mu+mu4e+exim to read+index/search+send my E-Mail. It works, and overall I'm happy with it.
I used to use mbsync but when my employer migrated from Cyrus to Exchange IMAP I couldn't get mbsync to work with it, but OfflineIMAP worked just fine.
Now they've migrated to GMail, which works fine with both, and even though OfflineIMAP is more flexible & easier to customize I'll probably switch back at some point.
OfflineIMAP regularly pegs my CPU at >100% just downloading a few dozen E-Mails, whereas mbsync's CPU usage never went above 2-3%. When my laptop fan starts spinning it's usually OfflineIMAP in the background.
So if you're looking for an IMAP->Maildir retriever or backup solution consider trying out mbsync first.
It's a PAIN to switch between these programs, or even using the same data when switching between different servers (e.g. Cyrus -> Exchange). Inexplicably neither of them have some sort of reindexing mode where they can take an existing Maildir and reconstruct their local database from local/remote Message-Id's. They both use & maintain an opaque local database. That's why I haven't simply gone back to mbsync.
> So if you're looking for an IMAP->Maildir retriever or backup solution consider trying out mbsync first.
I'll second this: I used OfflineIMAP at first, but switched to mbsynrc after being unhappy with OfflineIMAP's performance. (It also seemed to hang semi-sporadically for me, which made it a pain to launch as a cron job.)
Lots of factors affect which tool is the better for a given user, of course, but I personally have been happier with mbsynrc.
Are you using the OfflineIMAP SQLite driver? If not, the maintainers did say it was more performant and reliable than using text files which was the default before version 7.
Yes, I use the SQLite driver, and use & regularly pull the latest version from the git, 7.1 right now.
I haven't dug into why it uses that CPU in any detail. Looking at its output / tcpdump of what it's doing it goes to >100% CPU for 10-30 seconds when it's finished fetching the status from the remote & is presumably in some tight inner loop comparing it with local data to decide what to do before it starts downloading messages, and then briefly spends a few more seconds at >100% after the messages are downloaded, presumably updating its local state.
When it's just issuing IMAP commands or downloading messages from the remote it sits at 1-5% CPU.
As I pointed out OfflineIMAP works just fine for me, but in retrospect I'd be happier if I had stayed with mbsync. I don't need any feature from OfflineIMAP that it doesn't have, and it both uses less CPU/Memory & it finishes deciding what to download & downloading it in way less time.
I have just checked how much it uses on my side: doesn't go above 50%, but I do consider that pretty high and can totally understand how it could use 100% with bigger mailboxes. For me, this was never a problem since I only run it once daily for backup.
Thank you for mentioning mbsync, I will keep it in mind.
I'm in a situation similar to the one you were in (mu4e, Exchange IMAP at work), and mbsync (version 1.2) works for me. Not sure what Exchange version is there, and I had to configure it a bit both initially and after an MS Exchange update -- but just some basic settings, such as `SSLType STARTTLS`, setting CertificateFile manually, `AuthMechs LOGIN`.
I like mu4e because it integrates with orgmode well. However, it is not the best client for managing Exchange meeting invites and plain text responses are not very readable in outlook. Were you able to solely rely on mu4e without using Outlook?
Yes, and I've actually never used Outlook with Exchange. But I'm only sending plain text responses, not trying to use any advanced features, and it works fine to receive and read meeting invites (just as messages).
What goes wrong with plain text responses in Outlook, by the way?
Thirteen years of thunderbirding - downloading all imap for local keeping, and with numerous migrations between servers - has not lost me a single email. In fact, I have all mail correspondance since the mid nineties - around a hundred thousand mails - neatly archived in Thunderbird folders.
Me too. There is one caveat, though: I've found TB's UI starts to struggle when dealing with folders with a large number of messages, particularly on Linux. It's not uncommon for the UI to freeze up entirely when I switch the view from one folder to another; usually this just means having to wait a second or two for TB to become responsive again, so it's not the end of the world, but it is annoying.
fetchmail or getmail? I once used a very-slightly customized version of getmail (the Charles Cazabon program) in production to sync customers' GMail accounts into Salesforce.com, and it worked great (continues to work great, AFAIK).
I wanted to use an IMAP sync but after much experimenting with existing tools concluded that the protocol wasn't flexible enough for a proper two-way synchronisation.
I can't remember the exact reasoning now but several years on I am using dovecot's excellent dsync protocol over SSH to keep my laptop in sync with a private mail server, and have never looked back.
Combined with pine, exim and the excellent mairix (surprised not to see that mentioned yet) I enjoy a decent search on a fast MailDir implementation, with full offline support.
For those of you using GMail I'll repost a comment of mine [0] about gmvault [1]:
I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this:
I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different databases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.
> IMAP's main downside is that you have to trust your email provider to not lose your email. While certainly unlikely, it's not impossible. With OfflineIMAP, you can download your Mailboxes and make you own backups of your Maildir.
A professional mail host ('professional' as in 'someone that somehows makes money by hosting email') is way more unlikely to lose data than most of us are.
Making backup is great by the way, but for such task I'd strongly recommend the good old fetchmail or the newer (and better) fdm (https://github.com/ft/fdm). Both support imap(s). Both can save to either mailbox or maildir.
If you need a local copy, go for fdm and a local copy of your mailbox, plain and simple.
> A professional mail host ('professional' as in 'someone that somehows makes money by hosting email') is way more unlikely to lose data than most of us are.
I think their point is not that you're less likely to lose it yourself, it's that the probability that both you and the provider losing it at the same time is extremely unlikely. I.e., two points of failure required to lose your data.
The fdm project is dead. The Sourceforge project page returns 404, and the Github repository hasn't been updated since 2014. (And the Sourceforge mailing list returns 404 as well.)
The code probably works, but don't expect any help or updates.
A bigger worry with a professional mail host is they just cut your account for some arbitrary reason, and if that host is Google then good luck resolving the problem.
Would anybody please like to recommend a good sql schema (that makes sense) for storing emails (possibly downloaded with this fine piece of software). Also a hint about a maildir -> sql import tool would be interesting to see.
If you have already made your experiences with storing email in a sql database, it would be very appreciated if you would like to write about the pitfalls to avoid and the problems that occured.
I'd have a look at dbmail.org (an Imap server with a rdbms backend). If I've understood correctly, after some years they landed on a fairly straightforward schema with appropriate indexes.
Another question is what do you want to do? I've not played that much with sql store for mail - but notmuch (and mailutils) make good use of Xapian for quick index/search of mail, if that is your end goal.
This is good as a standalone IMAP client.
We have done a IMAP Client Server, which will run multiple IMAP accounts which means this particular server will fetch/sync all the mails and monitor for new mails and flag changes. Almost works with all the IMAP email server without any issue.
Some stats:
* We have more than 25K IMAP accounts running across 3 different 8core machine.
* Application is written over our custom async web server and our nio tcp client.
* Handling more than 50k connection per server instance.
Running a standalone client is much more easier because we have to concentrate only on syncing/fetching 1 account per instance. Rather in our IMAP Client Server we have to deal with multiple accounts with connection restrictions,failover handling, proper retry mechanism etc...
We have done this backend engineering for one our product.
Back when Fetchmail would segfault on certain spam messages, and a bug report to ESR went nowhere, I wrote my own mail fetcher in Perl. Then I extended it to IMAP, and later IMAPS, working around some bugs in some popular IMAP servers. I had it deliver mail to a local mda (so I could use Maildir and mda filters) and configurable to keep or delete mails from the remote server. I think it only supports one remote mailbox, however. https://github.com/psypete/public-bin/blob/public-bin/src/po... (It's very old code that sucks, and not secure, fwiw)
I use imapfilter[0] to sort imap mails on the remote server and move older ones to a local dovecot instance for long term backup. That way I have current mail available on all my devices and access to the backup via mutt or via file in the MH format on the backup system.
That works, but you have to program a bit lua in order to setup imapfilter correctly.
If i backup my Macbook hard drive (thus including all Mail data/my imap mail account) does this backup include ALL the data stored at my imap mail server? Or would i be missing certain parts, like attachments or whatever?
Would OfflineIMAP download 'more' data than i already have backed up?
I just installed nylas mail to see whether it support
well plain text emails.
It seems to me all this mail solutions requires that
the password of the mail server must be somewhere
visible in a poor plain text file which is a no-go
for me.
It seems to me all this mail solutions requires that the password of the mail server must be somewhere visible in a poor plain text file which is a no-go for me.
There's an addon that lets you store Thunderbird's passwords with Gnome's keyring. That said, in the end you must either type it each time or it must be stored in a way that can be read in an automated way.
Delightful. There are several ways to deal with it. I do this:
- For READING mail, my offlineimap knows about multiple accounts and puts them in different directories on disk
- mu indexes the PARENT of all those directories, so as far as mu is concerned, there's a single mail store with ALL the mail. The m: filter can be used to select specific directories, which map to different accounts here
- For SENDING mail, mu4e knows about multiple contexts that are detected based on rules you can set (for instance, when replying to email, it'll set the context to whichever account the mail I'm replying to was sent). Or one can set the context explicitly with the ';' binding.
OfflineIMAP is not a graphical email client. It is simply a tool with which you can either backup emails from your provider using IMAP or sync a local Maildir with your provider. The tool uses the Maildir email format. More information: https://en.wikipedia.org/wiki/Maildir
I used to use mbsync but when my employer migrated from Cyrus to Exchange IMAP I couldn't get mbsync to work with it, but OfflineIMAP worked just fine.
Now they've migrated to GMail, which works fine with both, and even though OfflineIMAP is more flexible & easier to customize I'll probably switch back at some point.
OfflineIMAP regularly pegs my CPU at >100% just downloading a few dozen E-Mails, whereas mbsync's CPU usage never went above 2-3%. When my laptop fan starts spinning it's usually OfflineIMAP in the background.
So if you're looking for an IMAP->Maildir retriever or backup solution consider trying out mbsync first.
It's a PAIN to switch between these programs, or even using the same data when switching between different servers (e.g. Cyrus -> Exchange). Inexplicably neither of them have some sort of reindexing mode where they can take an existing Maildir and reconstruct their local database from local/remote Message-Id's. They both use & maintain an opaque local database. That's why I haven't simply gone back to mbsync.