
OfflineIMAP: sync and backup tool for IMAP - TobyGiacometti
https://github.com/OfflineIMAP/offlineimap
======
avar
I use OfflineIMAP to fetch & emacs/mu+mu4e+exim to read+index/search+send my
E-Mail. It works, and overall I'm happy with it.

I used to use mbsync but when my employer migrated from Cyrus to Exchange IMAP
I couldn't get mbsync to work with it, but OfflineIMAP worked just fine.

Now they've migrated to GMail, which works fine with both, and even though
OfflineIMAP is more flexible & easier to customize I'll probably switch back
at some point.

OfflineIMAP regularly pegs my CPU at >100% just downloading a few dozen
E-Mails, whereas mbsync's CPU usage never went above 2-3%. When my laptop fan
starts spinning it's usually OfflineIMAP in the background.

So if you're looking for an IMAP->Maildir retriever or backup solution
consider trying out mbsync first.

It's a PAIN to switch between these programs, or even using the same data when
switching between different servers (e.g. Cyrus -> Exchange). Inexplicably
neither of them have some sort of reindexing mode where they can take an
existing Maildir and reconstruct their local database from local/remote
Message-Id's. They both use & maintain an opaque local database. That's why I
haven't simply gone back to mbsync.

~~~
wfleming
> So if you're looking for an IMAP->Maildir retriever or backup solution
> consider trying out mbsync first.

I'll second this: I used OfflineIMAP at first, but switched to mbsynrc after
being unhappy with OfflineIMAP's performance. (It also seemed to hang semi-
sporadically for me, which made it a pain to launch as a cron job.)

Lots of factors affect which tool is the better for a given user, of course,
but I personally have been happier with mbsynrc.

~~~
amirouche
There is the future version of offlineimap called
[https://github.com/OfflineIMAP/imapfw](https://github.com/OfflineIMAP/imapfw)

------
interfixus
Thirteen years of thunderbirding - downloading all imap for local keeping, and
with numerous migrations between servers - has not lost me a single email. In
fact, I have all mail correspondance since the mid nineties - around a hundred
thousand mails - neatly archived in Thunderbird folders.

For what it's worth.

~~~
tmalsburg2
If you have >100K emails, how do you know that nothing is missing?

~~~
devopsproject
It always says 100% complete when it runs /s

------
akie
I've successfully used imapsync
([https://github.com/imapsync/imapsync](https://github.com/imapsync/imapsync)
and [https://imapsync.lamiral.info/](https://imapsync.lamiral.info/)) in the
past. It's been around for years, has tons of options, and has proven very
stable, and is still maintained.

~~~
itsfatz
This saved me at least 2 hours of awful fetchmail/getmail related work last
month; Highly recommended.

~~~
finnh
fetchmail or getmail? I once used a very-slightly customized version of
getmail (the Charles Cazabon program) in production to sync customers' GMail
accounts into Salesforce.com, and it worked great (continues to work great,
AFAIK).

------
zbuf
I wanted to use an IMAP sync but after much experimenting with existing tools
concluded that the protocol wasn't flexible enough for a proper two-way
synchronisation.

I can't remember the exact reasoning now but several years on I am using
dovecot's excellent dsync protocol over SSH to keep my laptop in sync with a
private mail server, and have never looked back.

Combined with pine, exim and the excellent mairix (surprised not to see that
mentioned yet) I enjoy a decent search on a fast MailDir implementation, with
full offline support.

------
joshstrange
For those of you using GMail I'll repost a comment of mine [0] about gmvault
[1]:

I have been using this for years and it's great software. One tip, store each
different email account in it's own "database" my crontab looks like this:

    
    
        #Email backup
        0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume josh@joshstrange.com >> ~/mail-backup-logs/joshstrange.com.log 2>&1
        0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume other@otheremail.com >> ~/mail-backup-logs/otheremail.com.log 2>&1
        ....
    

I sync it all locally to my house then back it up to Dropbox as well. The
reason to store them in different databases is you cannot "filter" them out
when restoring so if they all go to the same DB if you restore you are
restoring ALL your email across all accounts to one new account.

[0]
[https://news.ycombinator.com/item?id=12987670](https://news.ycombinator.com/item?id=12987670)

[1] [http://gmvault.org/](http://gmvault.org/)

------
znpy
> IMAP's main downside is that you have to trust your email provider to not
> lose your email. While certainly unlikely, it's not impossible. With
> OfflineIMAP, you can download your Mailboxes and make you own backups of
> your Maildir.

A professional mail host ('professional' as in 'someone that somehows makes
money by hosting email') is way more unlikely to lose data than most of us
are.

Making backup is great by the way, but for such task I'd strongly recommend
the good old fetchmail or the newer (and better) fdm
([https://github.com/ft/fdm](https://github.com/ft/fdm)). Both support
imap(s). Both can save to either mailbox or maildir.

If you need a local copy, go for fdm and a local copy of your mailbox, plain
and simple.

~~~
jaskerr
The fdm project is dead. The Sourceforge project page returns 404, and the
Github repository hasn't been updated since 2014. (And the Sourceforge mailing
list returns 404 as well.)

The code probably works, but don't expect any help or updates.

~~~
znpy
I used it a couple of months ago and it worked perfectly.

Not all dead projects are actually dead. Some are just "completed".

------
JetSpiegel
See also the older but somewhat more reliably and faster isync AKA mbsync.

[http://isync.sourceforge.net/mbsync.html](http://isync.sourceforge.net/mbsync.html)

~~~
cannam
My first thought was "older? really?" \- but hosted at SourceForge, I guess it
could be.

I started using offlineimap in 2008, which feels like a long time ago but of
course is a lot more recent than the first appearance of imap.

~~~
jwilk
The first version of isync (0.1) was released in 2000.

The first version of offlineimap (1.0) was released in 2002.

------
softwarelimits
Would anybody please like to recommend a good sql schema (that makes sense)
for storing emails (possibly downloaded with this fine piece of software).
Also a hint about a maildir -> sql import tool would be interesting to see.

If you have already made your experiences with storing email in a sql
database, it would be very appreciated if you would like to write about the
pitfalls to avoid and the problems that occured.

Thank you very much for your attention!

~~~
e12e
I'd have a look at dbmail.org (an Imap server with a rdbms backend). If I've
understood correctly, after some years they landed on a fairly straightforward
schema with appropriate indexes.

There's also "Archiveopteryx": [http://aox.org/](http://aox.org/)

Another question is what do you want to do? I've not played that much with sql
store for mail - but notmuch (and mailutils) make good use of Xapian for quick
index/search of mail, if that is your end goal.

------
rakeshsr
This is good as a standalone IMAP client. We have done a IMAP Client Server,
which will run multiple IMAP accounts which means this particular server will
fetch/sync all the mails and monitor for new mails and flag changes. Almost
works with all the IMAP email server without any issue.

Some stats: * We have more than 25K IMAP accounts running across 3 different
8core machine. * Application is written over our custom async web server and
our nio tcp client. * Handling more than 50k connection per server instance.

Running a standalone client is much more easier because we have to concentrate
only on syncing/fetching 1 account per instance. Rather in our IMAP Client
Server we have to deal with multiple accounts with connection
restrictions,failover handling, proper retry mechanism etc... We have done
this backend engineering for one our product.

[https://www.zoho.com/salesinbox/](https://www.zoho.com/salesinbox/)

------
peterwwillis
Back when Fetchmail would segfault on certain spam messages, and a bug report
to ESR went nowhere, I wrote my own mail fetcher in Perl. Then I extended it
to IMAP, and later IMAPS, working around some bugs in some popular IMAP
servers. I had it deliver mail to a local mda (so I could use Maildir and mda
filters) and configurable to keep or delete mails from the remote server. I
think it only supports one remote mailbox, however.
[https://github.com/psypete/public-bin/blob/public-
bin/src/po...](https://github.com/psypete/public-bin/blob/public-
bin/src/pop3download/pop3download-0.8.pl) (It's very old code that sucks, and
not secure, fwiw)

~~~
jwilk
I guess fetchmail is more stable now. I've never seen any crashes, and I've
been using it against IMAP servers for at least a decade.

------
chme
I use imapfilter[0] to sort imap mails on the remote server and move older
ones to a local dovecot instance for long term backup. That way I have current
mail available on all my devices and access to the backup via mutt or via file
in the MH format on the backup system.

That works, but you have to program a bit lua in order to setup imapfilter
correctly.

[0]
[https://github.com/lefcha/imapfilter](https://github.com/lefcha/imapfilter)

------
DavideNL
If i backup my Macbook hard drive (thus including all Mail data/my imap mail
account) does this backup include ALL the data stored at my imap mail server?
Or would i be missing certain parts, like attachments or whatever?

Would OfflineIMAP download 'more' data than i already have backed up?

------
amirouche
I just installed nylas mail to see whether it support well plain text emails.

It seems to me all this mail solutions requires that the password of the mail
server must be somewhere visible in a poor plain text file which is a no-go
for me.

~~~
icebraining
_It seems to me all this mail solutions requires that the password of the mail
server must be somewhere visible in a poor plain text file which is a no-go
for me._

There's an addon that lets you store Thunderbird's passwords with Gnome's
keyring. That said, in the end you must either type it each time or it must be
stored in a way that can be read in an automated way.

[https://developer.pidgin.im/wiki/PlainTextPasswords](https://developer.pidgin.im/wiki/PlainTextPasswords)

------
dmd
For backing up GMail, can anyone compare this to
[https://github.com/gaubert/gmvault](https://github.com/gaubert/gmvault) ?

~~~
degenerate
Also, is there a similar tool to migrate from MSN/Yahoo to Gmail?

~~~
chrisper
There is.

[https://github.com/imapsync/imapsync](https://github.com/imapsync/imapsync)

------
urda
Sad to see:

    
    
        Python v3.4+ [STALLED]
    

But looks like development is pretty active so, hopefully they finish making
the jump to Python 3 soon!

------
consultutah
I need this and real full text search. Who will give me full text search? ;)

~~~
dima55
[https://notmuchmail.org/](https://notmuchmail.org/)

[http://www.djcbsoftware.nl/code/mu/](http://www.djcbsoftware.nl/code/mu/)

~~~
fusiongyro
Another +1 for mu/mu4e. Been using it at work for three or four years. Having
instant access to ~8 year old email is like a superpower.

~~~
znpy
question: how is mu/mu4e at handling multiple email accounts?

~~~
dima55
Delightful. There are several ways to deal with it. I do this:

\- For READING mail, my offlineimap knows about multiple accounts and puts
them in different directories on disk

\- mu indexes the PARENT of all those directories, so as far as mu is
concerned, there's a single mail store with ALL the mail. The m: filter can be
used to select specific directories, which map to different accounts here

\- For SENDING mail, mu4e knows about multiple contexts that are detected
based on rules you can set (for instance, when replying to email, it'll set
the context to whichever account the mail I'm replying to was sent). Or one
can set the context explicitly with the ';' binding.

Works quite well

------
orschiro
If I use this, is the result different than from using Thunderbird?

~~~
TobyGiacometti
OfflineIMAP is not a graphical email client. It is simply a tool with which
you can either backup emails from your provider using IMAP or sync a local
Maildir with your provider. The tool uses the Maildir email format. More
information:
[https://en.wikipedia.org/wiki/Maildir](https://en.wikipedia.org/wiki/Maildir)

~~~
badosu
Works great with mutt!

------
drkhsh
i use a similar program named isync aka mbsync. it is written in c and i like
it more than offlineimap!

------
AndyMcConachie
I believe this is called POP3.

------
j0hnM1st
Error: No available formula with the name "imapfw" :-(

~~~
TobyGiacometti
I don't think imapfw is widely adopted yet, it seems to be work in progress so
most people probably opt for the more stable predecessor OfflineIMAP.

