
Gmvault: Backup and restore your Gmail account - uyoakaoma
https://github.com/gaubert/gmvault
======
joshstrange
I have been using this for years and it's great software. One tip, store each
different email account in it's own "database" my crontab looks like this:

    
    
        #Email backup
        0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume josh@joshstrange.com >> ~/mail-backup-logs/joshstrange.com.log 2>&1
        0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume other@otheremail.com >> ~/mail-backup-logs/otheremail.com.log 2>&1
        ....
    

I sync it all locally to my house then back it up to Dropbox as well. The
reason to store them in different datebases is you cannot "filter" them out
when restoring so if they all go to the same DB if you restore you are
restoring ALL your email across all accounts to one new account.

~~~
laumars
Thank you for your advice, that's handy to know.

What's the database format it uses, sqlite3? (I tried looking on the repo but
couldn't any obvious reference)

~~~
leeoniya
if you want to try something that downloads Gmail via imap and indexes it into
an sqlite3 db (with FTS5 fulltext of from/to/cc/bcc/subject/body fields) and
extracts attachments to filesystem, take a look at a recent project of mine:

[https://github.com/good-3g-apps/email/tree/master/imap-
impor...](https://github.com/good-3g-apps/email/tree/master/imap-importer)

It also saves the raw .eml files to disk. No support for labels (yet), but it
does properly link up threads in the db using `References` from the parsed
headers (setting both MPTT and adjacency-list fields)

It's WIP, so contributions welcome :)

~~~
leeoniya
FWIW regarding speed, i was able to download, index & extract my entire INBOX
+ Sent Items (14k emails, 3.5GB total) in < 10min on a fast connection. the
limiting factor by far was connection/imap speed.

------
handruin
I'm interested to know what the pros and cons are of this utility vs using the
Google takeout functionality? I like the idea of this project but I don't know
what it would gain me over Google's native export? Is it the restoration
that's missing from Google's service?

[https://takeout.google.com/settings/takeout](https://takeout.google.com/settings/takeout)

~~~
xur17
My coworker and I both tried this yesterday, and today we both received an
email with this error message: "Sorry, we encountered a problem when creating
your Google data archive."

~~~
agumonkey
Took out my Google data a few times, now amounting 1.4GB never had this issue.
Let's hope it's only a temporary slipup.

btw, the data is pretty exhaustive, even for non major Google applications or
more recently added features (Maps Location History).

------
emilburzo
After yesterday's post[1], I've setup a cron job to regularly backup
Gmail/Google Drive.

I'm using Gmvault for Gmail (emails and chats), pretty effortless and with
enough features for me, especially the export to maildir.

And rclone[2] for Google Drive, in "create a local mirror" mode (sync).

Any other tools to backup your Google life?

[1] -
[https://news.ycombinator.com/item?id=12972554](https://news.ycombinator.com/item?id=12972554)
[2] - [http://rclone.org/](http://rclone.org/)

~~~
jdc0589
I am _actively_ trying to get off of dropbox. I really want a similar native
application experience that syncs to-and-from S3. Not a cron job using the s3
cli, not something that only works on osx/windows/etc... So far owncloud
enterprise is the only polished looking solution I've found, but thats a bit
overkill...

~~~
ReAzem
You are looking for Syncany:
[https://www.syncany.org/](https://www.syncany.org/)

But really, you may not need S3 at all and just sync between your devices with
Syncthing: [https://syncthing.net/](https://syncthing.net/)

Shameless plug: Syncthing is in official Debian repositories!
[https://tracker.debian.org/pkg/syncthing](https://tracker.debian.org/pkg/syncthing)

~~~
mahyarm
How have you found syncthing perf? When I tried it a while back the perf was
horrible. It ate CPU and was pretty slow, even though I was just testing with
2 machines on my local network.

~~~
ReAzem
This was improved recently with 14.0.7:

\-
[https://github.com/syncthing/syncthing/releases/tag/v0.14.7](https://github.com/syncthing/syncthing/releases/tag/v0.14.7)

Also, It will improve further with File System Notifications (from what I
understand?):

\-
[https://github.com/syncthing/syncthing/pull/2807](https://github.com/syncthing/syncthing/pull/2807)

------
wfunction
WARNING/fun fact: it doesn't download all emails properly. Last time I tried
it, it seemed that when the Gmail server randomly closed a connection (or
maybe some other time, but I think it was in these instances), the program
would just keep whatever partial results it had and then move on to the next
email. Which meant I had a lot of partial emails on my drive (only a small
fraction of all the emails, but still), and no way to detect them.

------
ljosa
I have been using both GYB ([https://github.com/jay0lee/got-your-
back](https://github.com/jay0lee/got-your-back)) and mbsync
([http://isync.sourceforge.net/mbsync.html](http://isync.sourceforge.net/mbsync.html))
for this. How is gmvault better/different?

~~~
wielebny
I've been using gmvault for only one feature: ability to fetch e-mails that
match a certain filter.

~~~
chrislund
GYB does that, too: [https://github.com/jay0lee/got-your-back/wiki#selective-
back...](https://github.com/jay0lee/got-your-back/wiki#selective-backups-with-
gmail-searching)

------
pawadu
Given how easy and carelessly Google can close your account and ruin your
digital life, I guess periodic backups of your cloud accounts will soon be
considered a good practice.

~~~
scholia
_> I guess periodic backups of your cloud accounts will soon be considered a
good practice._

It has been good practice for a couple of decades. Of course, getting people
to follow good practices is another matter....

~~~
bartvk
In this case, it's not as good as a normal backup. To my knowledge, Google
Docs/Sheets/etc. is exported in other file formats (MS Word format, PDF
format, etc), not their internal file format. So it's not a real backup, just
an export.

~~~
scholia
Good point. And we have no way of knowing what Google's internal format looks
like...

------
bit_logic
Data is only half the problem. The other is that email address has become the
"primary key" for everything. Banking websites, random forums, everything.
It's sometimes impossible to change it because it's the "primary key" for
identity on that website. And email addresses are not portable like phone
numbers.

~~~
eigenvector
Email addresses are portable to any webmail, mail server or other email
infrastructure provider if you have your own domain and then forward to the
service of your choice. This way you can use Gmail if you like or maintain
your own full email stack, or anything in between, while still addressing your
mail to something you control.

~~~
tetraodonpuffer
but do you really control your domain? if somebody hacks into your registrar
or forges your signature or something and transfers it to themselves, they
would get your email, and it might be very difficult for you to get it back.
Or maybe I am being too paranoid about this and it's less likely that your
domain gets taken from you compared to your mail provider deciding to
terminate you.

You can also do something like using fastmail, where you don't control your
domain, but being a paying customer you do have somebody to call if there are
issues

~~~
icebraining
I find the attack itself technically plausible, but where's the motivation?
Why would someone go to the effort just to get some random personal domain?
Sounds a little far-fetched, and I've never heard it happening to a personal
domain of a non-VIP. Meanwhile, we know that Gmail accounts get terminated
regularly, and appealing is hard-to-impossible.

~~~
xyzzy123
Cryptocurrency mostly (based on public disclosures), but sometimes other
digital assets like twitter handles or game currency. Sometimes domain names
themselves are the asset.

As is now well known, mobile numbers are often also hijacked to subvert weak
2fa.

------
laurent123456
Since Gmail supports imap and pop3, one could simply use any proven email
client to backup the emails. I don't think a special tool that or may not work
is needed for this.

~~~
cookiecaper
Which "proven email client" do you recommend? There's an Import/Export
extension for Thunderbird, one of the last standing desktop mail apps, but
it's not good at handling huge, multi-thousand message exports.

gmvault is purpose-built and quite simple. I've used it before with success.
It backs up your whole mailbox in one command. Why should we fiddle with a
desktop mail client?

~~~
pi-rat
fetchmail, getmail and offlineimap are all "proven email clients" that work
great for this.

~~~
Twirrim
+1 on offlineimap. Those of us that may be a little strange in the head and
prefer using mail clients like mutt will often use it to handle background
imap syncing.

------
AdmiralAsshat
The 'restore' feature looks nice. But would it be of any value if Google
decides to close your account? That is, can I take my backup emails from Gmail
and 'restore' it to myemail@self-hosted-domain.com in order to migrate all
those emails over?

~~~
timdafweak
Good question, I wonder if anyone’s tried this before. It would be invaluable
if that were the case.

~~~
whyagaindavid
I have opted to have 3 gmail accounts with various names; 1 outlook, 1 yahoo
account. Every email sent to my gmail is forwarded > outlook > yahoo. I use 2
other gmail accounts for newsletters and etc. I am sure at least 1 company
will remain free to access data! BTW thanks for the thunderbird tip.

------
kingnight
My backup of Gmail is to use Mail.app via IMAP + download all attachments.

I then have a backup of my computer -> 1\. Time Machine + 2\. Arq->S3/Glacier

Given that this keeps mail locally in a constantly readible format (offline,
copied in mbox)... is there something missing in my basic solution that this
cli utility adds?

~~~
jxy
Or any IMAP client that saves messages locally.

I guess this utility benefits people who only use gmail's web interface.

------
ajennings
Here's one I wrote: [https://github.com/abjennings/gmail-
backup](https://github.com/abjennings/gmail-backup)

Not as full featured (can't restore), but it's just a 77-line Python script.
You could audit it yourself to make sure it doesn't upload your creds to
another server.

------
lexalizer
Please don't forget to encrypt your backups in case your local drive gets
stolen or your cloud backup service is hacked.

------
dharma1
doesn't [http://www.google.com/takeout](http://www.google.com/takeout) include
gmail?

~~~
cdubzzz
Yep! And much more. But gmvault does also restore emails to an account,
interestingly.

I am (slowly) working on a project to pull some statistics from Takeout's mbox
file for Mail. Also want to play around with the Location History, Chrome data
and Hangouts exports.

~~~
pokemongoaway
That would be very useful. I'd like to use takeout incrementally - and have
some tools to use those backups at any time (like you describe).

------
hengheng
Since yesterday I have a few Google Takeout zip files in my backup (
[https://takeout.google.com/settings/takeout](https://takeout.google.com/settings/takeout)
). I've used gmvault in the past, but this looks superior from the outside.
Haven't delved into the data I'll admit.

~~~
cdubzzz
I am working on digging around this data right now, actually. Some details on
my website (see profile) and I will post on HN about it at some point when I
get things further along (also want to look at Location History and Chrome
data exports).

------
amq
Why should I use this over an email client like Thunderbird?

~~~
tajen
Do you use IMAP? Do emails get fully downloaded in IMAP?

~~~
detaro
You can set it up like that, yes. (I'm not sure what the default is, since I
haven't created a new Thunderbird profile in years)

------
hatchoo
I have used this in the past to backup email accounts of resigned employees so
we continue to stay well within the maximum number of active accounts for our
free Google Apps.

It generally works fine and it allows you to restore the emails to a different
account name. (I sometimes temporarily restore an account to search for old
emails). It seems to have some issues with restoring accounts with a lot of
large emails (large or multiple attachments) especially those that have
reached the 15GB quota.

------
hobarrera
> Handle all Gmail IMAP hiccups.

It's sad that after all these years, google still hasn't gotten around to
properly implementing IMAP.

------
fgpwd
Is it considered safe to leave your oauth ID and Secret public in open source
projects?

------
ajonit
I have been using Spanning backup for several years now. It backs up our
Gmail, documents , calendars, contacts, sites It is one of those set-it-and-
forget-it type services. The company is acquired by EMC now. The service costs
us around $35/year/email account; for us it is small price for peace of mind.
I wrote a review of them few years back -
[http://reviewofweb.com/gmail/backupify-vs-
spanning/](http://reviewofweb.com/gmail/backupify-vs-spanning/)

------
aq3cn
I have been planning to switch to mutt, this tool will also come in handy.

Are there similar tools for MS Outlook or Protonmail or if it is possible to
modify gmvault to work with either one?

------
josefresco
I use Thunderbird Portable to backup our Google Apps/GSuite email accounts.
Open once a week - Bob's your uncle.

------
tonyplee
Does it work on Raspberry Pi?

I have an always on PI server as git server. I wouldn't mind put this on as
one more cron job on it.

~~~
icebraining
It's written in Python, so it should work.

------
hmart
Seems legit. In the past I have used Imapsync to transfer hundreds of email
accounts out of gmail / google apps.
[https://imapsync.lamiral.info/](https://imapsync.lamiral.info/)

------
honkhonkpants
Ironically using some of these buggy tools can be an effective and swift way
to get your account locked out or at least rate limited by anti-abuse systems.
Read the code and make sure they are using official APIs and are coded sanely.

~~~
tomschlick
They are. The first thing gmvault does is have you authenticate with oauth.
From there it's just using the gmail apis / imap to download the data.

------
tscs37
Best timing for posting this.

Even before this I've been looking to get a bit independent from Google, atm
I'm trying to get Camlistore to replicate my data over various places. (I wish
it had some erasure encoding)

------
Roritharr
Are there similar tools for google docs documents? drive files? contacts?

~~~
tomschlick
For drive:
[https://github.com/prasmussen/gdrive](https://github.com/prasmussen/gdrive)

------
joshfraser
A lazy alternative is to forward all incoming email to a yahoo account. I've
been doing this for years and it's come in handy on the rare occasions when
Gmail is unavailable.

------
andydoan
Does this only backup attachments and images as well as text? I can't seem to
find any reference to this on the site.

EDIT: Ok it appears gmvault encodes attachments into the eml default, cool.

~~~
prplhaz4
I just searched this thread, the site, and github...Here's the best I've
found...apparently it supports saving attachments...

[https://github.com/gaubert/gmvault/issues/73](https://github.com/gaubert/gmvault/issues/73)
>> Users asked about attachments. Put somewhere in the doc that they are
saved.

------
kylehotchkiss
I was trying to push some people off my Google Apps accounts since google
won't allow you to "split" some users off the account. Wish I would have seen
this then!

------
SadWebDeveloper
Offtopic: I have Gsuite (free) for my company (<50), is it possible to backup
all the emails of my users without knowing their passwords (with XOauth
tokens?).

------
equalunique
Cool, but I'm going to have to do a source code review before I trust any new
tool with my Gmail messages.

------
rrggrr
No Python 3 support (at least under Anaconda), but interested when its
supported.

~~~
icedchai
Why does this matter? Just set up a virtualenv. I've been using this to backup
my gmail account for years.

~~~
rrggrr
Excellent idea, never occurred to me. -1 for snark, +2 for coming to my aid.

------
falsedan
How does this persist/restore filters?

------
jrcii
I use mutt setup according to Steve Losh's advice "The Homely Mutt" to achieve
this with great success.

------
mikebay
Yes, we need to backup our emails before google delete our accounts. Googles
operations start to remains me of obamas administration, where nobody can
actually say anything out of template..

