
IMAP Email Backup - wheresvic4
https://smalldata.tech/blog/2020/04/07/imap-email-backup
======
scandox
Recommend IMAPSync:
[https://imapsync.lamiral.info/](https://imapsync.lamiral.info/)

Have used it at pretty massive scale and it handles a hell of a lot of
different options.

~~~
buro9
This one... and I used it to migrate 20 years of email from a G Suite account
to Gmail account.

[https://medium.com/@buro9/one-account-all-of-
google-4d292906...](https://medium.com/@buro9/one-account-all-of-
google-4d2929066951)

~~~
BuildTheRobots
You freaked me out with the 20 years - I didn't think it'd been _quite_ that
long. 2006 as "Google Apps for Your Domain", in case anyone else is wondering.

~~~
buro9
In my Gmail I have more than 20 years, all email I've sent and received since
1999. I've maintained a single source of emails and move it forward as I've
changed provider.

------
akerro
Thunderbird is your desktop client that can archive, compress, and store all
emails offline. I've even used Thunderbird for migrating all my email history,
folders etc. between servers: simply add new account and drag and drop folders
between accounts. Simple enough to be used by non technical people who do not
want to:

>./imapgrab.py -d -v -f ~/user@email.com -s imap.mail.server -S -p 993 -u
user@email.com -p password -m
"_ALL_,-INBOX.Trash,-INBOX.Spam,-INBOX.Junk,-Trash,-Junk,-Spam"

~~~
bklaasen
Be careful using Thunderbird to move mail between IMAP4 servers, on Linux at
least - I've had it lose messages on me, deleting them from the source before
they're successfully written to the destination.

I was moving mail from a Gmail account to a Gandi account about a year ago,
before the recent spurt of development activity on Thunderbird.

~~~
JonathonW
The only time I've ever lost mail was while trying to move a bunch of messages
in Postbox (shares Thunderbird's backend). Wasn't even moving between
servers-- this was just a folder-to-folder move.

Always copy, verify that the messages really are at the destination, then
delete. I don't know why this isn't what Thunderbird does on its own when you
move mail, but it isn't.

------
teddyh
I use OfflineIMAP:
[https://www.offlineimap.org/](https://www.offlineimap.org/)

~~~
aidenn0
I also use offlineIMAP, but I find that every few months it freezes and needs
a kill -9 to exit. My current workaround is killing it once per week and
letting my service manager restart it.

------
suchoudh
I never was able to backup emails from gmail successfully ~ 10GB of mails
right now. Everytime it fails somewhere in between because of reasons
mentioned by various people elsewhere in this thread.

Every year I try and fail.

This year with the comments herein I again tried and get invalid credentials
error with my Gmail.

1) Used mutt/getmail/python script ( password is correct verified it )

2) Even changed password and still same issue

3) I also tried to enable less secure apps to access email option in google
settings. But keep getting the Authentication error. Can someone please point
me to a good note on how to get Authentication error sorted ( maybe more to
take care of ) ( I use ubuntu bionic , python 2.7.17, getmail 5.5 )

Error rahu@rahu:~/0del/_working/_backupEmail$ ./ss.sh IMAP Grab 0.1.4 \---
List option selected Connecting to IMAP server via SSL Logging into IMAP
server Traceback (most recent call last): File "./imapgrab.py", line 444, in
<module> imapgrab() File "./imapgrab.py", line 150, in imapgrab ig_list =
IG_list_mailboxes(ig_options) File "./imapgrab.py", line 193, in
IG_list_mailboxes ig_imap.login(ig_options.username,ig_options.password) File
"/usr/lib/python2.7/imaplib.py", line 523, in login raise self.error(dat[-1])
imaplib.error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

~~~
theandrewbailey
Have you tried Google Takeout?

[https://takeout.google.com](https://takeout.google.com)

------
dijit
In the same vein the push towards not owning your data also comes with
issues[0].

I get along well with a tool called offlineimap[1] though it's python2; you
can even use it to sync imap backends[2]. I use it simply to populate a
Maildir on my machines.

Sadly I have not found it to work with office365 (then again, what does other
than outlook and thunderbird's proprietary subscription based extension?).

[0]: shameless plug: [https://blog.dijit.sh/importance-of-self-hosted-
backups](https://blog.dijit.sh/importance-of-self-hosted-backups)

[1]:
[http://www.offlineimap.org/doc/quick_start.html](http://www.offlineimap.org/doc/quick_start.html)

[2]:
[http://www.offlineimap.org/doc/backups.html](http://www.offlineimap.org/doc/backups.html)

~~~
404-universe
Hmm, my school uses office365 for its email, and I've managed to successfully
hook it up to offlineimap for my email. I think the only real "special" thing
that I had to do was have it use my email address (i.e. user@university.edu)
as my username to authenticate with.

The relevant section of my .offlineimaprc (trimmed for brevity, edited for
privacy):

    
    
      [Repository remote]
      type = IMAP
      remotehost = outlook.office365.com
      remoteuser = "user@university.edu"
      remoteport = 993

~~~
dijit
I think the admin might have enabled some compatibility features for you, I
get:

    
    
        ~ $ offlineimap
        OfflineIMAP 7.3.2
          Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
        imaplib2 v2.101 (bundled), Python v2.7.18, OpenSSL 1.1.1g  21 Apr 2020
        Account sync work:
         *** Processing account work
         Establishing connection to outlook.office365.com:993 (work-remote)
        Enter password for user '<user>@massive.se':
         PLAIN authentication failed: AUTHENTICATE failed.
         LOGIN authentication failed: LOGIN failed.
         ERROR: All authentication types failed:
         PLAIN: AUTHENTICATE failed.
         LOGIN: LOGIN failed.
         *** Finished account 'work' in 0:11
        ERROR: Exceptions occurred during the run!
        ERROR: All authentication types failed:
         PLAIN: AUTHENTICATE failed.
         LOGIN: LOGIN failed.
        
        Traceback:
          File "/usr/lib/python2.7/site-packages/offlineimap/accounts.py", line 293, in syncrunner
            self.__sync()
          File "/usr/lib/python2.7/site-packages/offlineimap/accounts.py", line 369, in __sync
            remoterepos.getfolders()
          File "/usr/lib/python2.7/site-packages/offlineimap/repository/IMAP.py", line 452, in getfolders
            imapobj = self.imapserver.acquireconnection()
          File "/usr/lib/python2.7/site-packages/offlineimap/imapserver.py", line 586, in acquireconnection
            self.__authn_helper(imapobj)
          File "/usr/lib/python2.7/site-packages/offlineimap/imapserver.py", line 459, in __authn_helper
            "failed:\n\t%s"% msg, OfflineImapError.ERROR.REPO)

------
memset
I wrote a small tool in go which allows you to delete email in bulk by sender.
It uses the gmail api and conforms to rate limits. It’s pretty neat! Slow,
perhaps, the first time you run it since it fetches threads one at a time.

[https://github.com/poundifdef/gmail-
deleter](https://github.com/poundifdef/gmail-deleter)

The next step would be to have it save and archive email threads. (Would this
be useful to anyone?)

------
sneak
Reminder: You likely don't need your email provider to store years and years
of old emails. Download them locally, then remove them from the service
provider.

[https://en.wikipedia.org/wiki/Third-
party_doctrine](https://en.wikipedia.org/wiki/Third-party_doctrine)

Thanks to the third-party doctrine, messages stored on the provider can be
disclosed to the government without a warrant. Your emails have contact
history, location histories, attachments, plans, all sorts of stuff.

Don't leave it on the server, where it's available for the taking. Even a
password breach will deliver the entirety of it to an attacker.

~~~
wheresvic4
Honestly, I'm a real proponent of keeping a clean digital life. I practice 0
inbox, aggressively delete old documents, photos on a regular basis. If I
haven't touched anything in a few years, the chances that I'm going to need it
later on down the line are slim and so off they go.

Important documents like tax info etc, are obviously kept for the minimum
legal period but then again they don't take up all that much space.

I've been doing this for around 10 years now and so far I have not had any
problems with missing old stuff, _touchwood_.

Long story short, just delete stuff you don't need altogether, makes backups a
breeze and frees you from worrying about hackers exposing your musings from
when you were a teenager :)

~~~
sneak
I have a 900GB FLAC music library, because cloud music services can disappear
tracks down the memory hole from the middle of your playlists your now-dead
loved ones made for you years ago[1], and also because they simply don't have
a lot of the latest/newest interesting/underground music in their cloud
catalogs.

Also:

    
    
        nostromo:~/Pictures$ du -sh LightroomMasters
        1.5T LightroomMasters
        nostromo:~/Pictures$ find LightroomMasters -type f | wc -l
          207655
        nostromo:~/Pictures$ ls -1d LightroomMasters/* | sort -n | head -1
        LightroomMasters/1997
        nostromo:~/Pictures$
    

I've been taking photographs of events in my life for a long time, and I look
at them regularly, which brings me joy. Sometimes I send loved ones photos I
took of them 3, 5, 10, 15 years ago that they've never seen, and it brings
them joy as well.

There are different paths, and digital minimalism is only suitable for a
subset of the population. Many of us derive immense value from having high
fidelity records of our lives and the lives of those close to us.

Also, if you store your historical data for a long time, you can do super cool
shit like this: [https://writings.stephenwolfram.com/2012/03/the-personal-
ana...](https://writings.stephenwolfram.com/2012/03/the-personal-analytics-of-
my-life/)

[1]: In case you can't tell, this happened to me.

------
igammarays
I use DevonThink (Mac-only). In addition to continuous archiving of email, it
indexes and stores attachments, files all my receipts, and keeps my PDFs and
archived notes there. Very powerful and reliable search.

[https://www.devontechnologies.com/apps/devonthink](https://www.devontechnologies.com/apps/devonthink)

------
al8ieK4o
I've been using a similar setup for a few years, and can totally vouch for it.
Emails are probably one of the most important things to backup. Not sure what
`imapgrab` adds, since fetching mails from imap is exactly what getmail is
doing.

That being said, if you go on and set up getmail, I'm not sure what's the
point to continue to use mutt through imap (the author mentions decrypting the
archive before reading them, so I assume it's not their daily driver and they
keep using direct imap connection otherwise). Imap is still useful if you want
to read mails on your mobile.

For the encryption part, I found cryfs [1] to be very useful. It's a fuse
based app that encrypts files on the fly as they are written to the pseudo fs
and stores them in partioned small size encrypted files. Basically, it means
you can use the pseudo fs as a regular directory, store your mail in it and
read them from it, and then the special layout of the encrypted directory with
its small files is especially rsync-friendly (outside of email system, if I
store a big file where only a small part has changed, only the files related
to these small changes will be uploaded).

An other advantage of going full local : you can use notmuch [2]. It's a mail
indexer that allows fast and advanced search within mails. I have all my mails
since 2013 locally, and it takes less than a second to make a full text search
within them.

The last thing that I love is the ability to use maildrop [3]. It's regexp
based filtering and can forward the mail to anything (mailbox or program), so
it allows to have advanced filters that can do quite crazy things (my phone
sends me mails when my battery is depleted or fully charged, and it triggers
desktop notifications on my computer). Not to be abused, obviously. Don't
allow anyone to shut down your computer by sending a email :)

[1] [https://github.com/cryfs/cryfs](https://github.com/cryfs/cryfs)

[2] [https://notmuchmail.org/](https://notmuchmail.org/)

[3] [https://www.courier-mta.org/maildrop/](https://www.courier-
mta.org/maildrop/)

------
Zhyl
First: I use neomutt + mbsync (formerly isync) + notmuch. The local searching
was at first a nerd special interest project but has actually become very
useful for searching, filtering and otherwise managing my mail.

Second: the button on the article to submit to HN seems to be somewhat... off?
I wouldn't mind one that would submit in the first instance and then link to
the HN thread on subsequent clicks but having a button that just loads a round
into the chamber _for all users clicking on it_ seems like incitement to
blogspam.

~~~
wheresvic4
The issue with local search is that you anyways end up doing search on the
server when on mobile or when you are not on your home computer. But yes,
definitely useful. I personally do 0 inbox so I hardly have anything to
search, heh.

Regarding the submit to HN button: if the article has already been posted, HN
will itself just link you to it rather than make a new submission. So it is a
bit of an intelligent hack where it piggybacks on HN's own code, hope that
assuages you :)

~~~
Zhyl
>it piggybacks on HN's own code, hope that assuages you :)

Haha, it does actually. That is very cunning!

------
rcarmo
Hello there,

I've maintained
[https://github.com/rcarmo/imapbackup](https://github.com/rcarmo/imapbackup)
since 2013 (and used early versions of it for years before that), so feel free
to try it out and see if it fits your use case (mine was snapshotting existing
IMAP mailboxes into mbox format, which was what we could use for migrating
mailboxes across systems then, and also fit my personal use case for backups).

~~~
miles
Thanks so much for crafting and maintaining imapbackup, Rui! It worked a treat
in backing up tens of thousands of messages that Mail.app refused to archive
properly.

The feature set is likely to appeal to many HN users:

* ZERO dependencies. (other than Python 2.5 or higher)

* Copies every single message from every single folder (or a subset of folders) in your IMAP server to your disk.

* Does incremental copying (i.e., tries very hard to not copy messages twice).

* Tries to do everything as safely as possible (only performs read operations on IMAP).

* Generates mbox formatted files that can be imported into Mail.app (just choose "Other" on the import dialog).

* Optionally compresses the result files on the fly (and can append to them).

* Is completely and utterly free (distributed under the MIT license).

------
jpeeler
If you're a gmail and a notmuch user, consider using lieer -
[https://github.com/gauteh/lieer](https://github.com/gauteh/lieer). It uses
the Gmail API to synchronize mail bidirectionally so that you can handle mail
locally (except for muting threads).

If any Googler is reading this, why oh why is muting not part of the API?

------
qhwudbebd
The tool I used to use for this years ago is isync/mbsync, which worked really
well. Nowadays I just use rsync on Maildirs because I don't have any IMAP mail
on servers where I don't have shell access, so I'm a bit out of date what the
modern state of these tools is. Interesting to hear of a variety of other
options in this thread that I'd not come across back then.

~~~
al8ieK4o
> Nowadays I just use rsync on Maildirs because I don't have any IMAP mail on
> servers where I don't have shell access, so I'm a bit out of date what the
> modern state of these tools is. Interesting to hear of another option I'd
> not come across back then

If you use rsync through ssh, I would say that's a very good option, if
anything because on your server, you only need ssh to be exposed to the
internet rather than ssh + an imap server. Given how servers are constantly
attacked, this is way safer.

------
deeblering4
imapsync is my personal favorite approach to this problem.

In addition to imap -> filesystem backups it supports synchronizing mail in
both directions. And both source and destination can be remote systems.

~~~
doesnotexist
The imapsync page
[https://imapsync.lamiral.info/](https://imapsync.lamiral.info/) states
"imapsync is not suitable for maintaining a synchronization between two active
imap accounts while the user is working on both sides. Use offlineimap
(written by John Goerzen) or mbsync (written by Michael R. Elkins) for
bidirectionnal (2 ways) synchronizations."

------
znpy
Shout out to office365 for being a piece of crap when it comes to IMAP.

If you launch something like fetchmail or imapsync at it, it will close the
connection in the middle of the sync because you're doing something nasty in
his opinion.

Last time I switched jobs and had to download 18+ GB of emails it took me days
to download them all off office 365, and I was to run it in a while true loop
in screen. Luckily I didn't wait for the last day to start downloading my
emails.

~~~
jgalt212
> Luckily I didn't wait for the last day to start downloading my emails.

And also luckily, your company didn't sue you or try to have you arrested for
extracting 18+ GB of company property.

~~~
znpy
That is also 18+ gb of my life.

~~~
sandeeps_
wait, you use work email for personal stuff??

~~~
znpy
No, but those years of work are part of my life.

I mean: my colleagues and I faced countless issues, we had both problems and
celebrations, and I want to be able to remember all of that in the future.

------
_salmon
I've used MailStore Home[1] for a few years and I really like it. It allows me
to search across all of my mailboxes, even ones that have since been shutdown,
for emails that I only remember a partial subject for.

[1]: [https://www.mailstore.com/en/products/mailstore-
home/](https://www.mailstore.com/en/products/mailstore-home/)

------
cowmix
For gmail, gmvault is great:

[http://gmvault.org/](http://gmvault.org/)

------
suhastech
I was in a similar situation a few years ago and decided to build an app for
that: [https://thehorcrux.com](https://thehorcrux.com)

My goal is to make it simple enough for my grandpa to use it. However, I'm
still not there yet. Let me know what you guys think.

~~~
notwhereyouare
just a bit of website critique. your menu wraps on my 1920x1080 screen.

Contact wraps to the next line

------
ovnz
Is there a way I can view such backups? Search in them? Something like offline
read-only maildir viewer?

I made a separate thread here:
[https://news.ycombinator.com/item?id=23425995](https://news.ycombinator.com/item?id=23425995)

------
peterrus
For those using Protonmail: I did a little writeup on using offlineimap to
keep a local backup.

[https://github.com/peterrus/protonmail-export-
linux](https://github.com/peterrus/protonmail-export-linux)

------
kccqzy
For me there's really no need to specifically backup my email. Instead, my
IMAP client is configured to download all messages and attachments to local
disk, and regular disk backups should take care of that.

------
jandrusk
I use offlineimap to sync mail locally and then periodically backup the mail
folders to an encrypted USB drive. Using Protonmail with ProtonBridge to pull
it down.

------
Angostura
Its the reason that I still use POP. Full stop.

~~~
rexreed
Same here. Happy POP user for decades. How do you deal with accessing mail
from multiple devices and when away from your main machine tho?

~~~
Angostura
I have my mailbox settings set to retain mail on the server for 30 days.

The fact that they are marked as unread from other machines is a pain, but not
insuperable.

------
nicolaslem
I have a cronjob on my NAS that runs mbsync every night. Really simple and
efficient.

------
RcouF1uZ4gsC
> python 2.7.x (does not work with python3)

Python 2 has a very, very long tail.

------
paololulli
fetchmail is the tool of the trade, and it does not need python.

