#offlineimap

[blueavenuejellyfish-blog]

Setting up mutt with offlineimap, msmtp (multi-account)

[gokhankici]

Add Gmail to offlineimap

First, write your gmail pasword to ~/.passwd/gmail and encrypt it with the command cd ~/.passwd && gpg -e -r '<your email>' gmail

Then create the following python file for reading gmail.gpg file that is created in the first step:

import os import subprocess def mailpasswd(acct): acct = os.path.basename(acct) path = "/home/<username>/.passwd/%s.gpg" % acct args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path] try: return subprocess.check_output(args).strip() except subprocess.CalledProcessError: return ""

Create your .offlineimaprc file like the following:

[general] accounts = Gmail maxsyncaccounts = 3 pythonfile = /home/<username>/bin/offlineimap.py [Account Gmail] localrepository = Local remoterepository = Remote [Repository Local] type = Maildir localfolders = ~/Maildir [Repository Remote] type = IMAP remotehost = imap.gmail.com remoteuser = <email> remotepasseval = mailpasswd("gmail") ssl = yes maxconnections = 1 realdelete = no sslcacertfile = /etc/ssl/certs/ca-certificates.crt

That's it! Now wait (probably for a long time) for offlineimap to download all your emails from the server :)

reference

[fimblo]

Mutt, Gmail and notmuch

When I started working at Spotify, I was overjoyed that the company allowed use of IMAP to synchronize mail. Since I was a mutt user, I set things up so that I could read and send mail using mutt when I was by my computer. When I was on the run I could always use the normal gmail interface.

For all of the inherent greatness of mutt, there was one thing which annoyed me - changing folders takes time. Say you're in your inbox, happily reading mail. Then you realize you want to check a mail you had sent earlier and you open your "sent" folder. This operation took quite some time; initially because the mail headers needed to be read from all the mails in the folder. Later on, with a patch that enabled mutt to save the headers, it was much quicker, but it still took time. 

Due to this I had changed my way of using mutt quite a bit - all my sent mail was saved in my inbox, for example, so that I could see what I had sent without changing folders. And in Gmail, I avoided archiving emails - since this would mean that I would need to look in multiple folders - one for new stuff and things I was working on right now, and another for all the archived stuff. 

Today, a colleague of mine (hi Lowe!) recommended a search tool for mutt - called mutt-notmuch-py, which ran using a indexer/search tool called notmuch.

In short, given that you imap-sync your email to your computer using some tool, it will create an index of your local mail, making it possible to do really quick searches in it. The results of the email search ends up in a temporary mutt folder, which is very nice - since you can then run your normal mutt operations on it like any other folder. Very useful.

I started playing with it, and though the indexing took quite some time (I have some 150000 emails in my inbox), once it was all indexed, I was overjoyed with the snappiness of the search operation. It was quick!

...and as I did all this, I started to feel a little uneasy. After a while I realized that this was because I didn't know how this would play with my "don't support multiple folders" rule.

So I started thinking about how this new shiny tool could fit my workflow... and came to the sudden thought that this was the tool I needed to ditch my old workflow completely!

How? Well, first off, if I archive all my old mails, the folders would be smaller. This means indexing operation will be faster. Added to this, nowadays I'm using a machine with a nice quick SSD drive - reading from these is much faster than on my old 5400RPM disk.

Mutt's limit function (think search with a vengeance) is really good - but only works on one folder. With notmuch, this would no longer be a problem - since it indexes all folders and presents the results in the same virtual folder. Nice!

So a few hours ago, there was only one thing stopping me from testing this new workflow - which is to actually archive my old emails. This is easier said than done, since in the past, Gmail would crash if you tried to do even the simplest operation on a larger batch of emails. One colleague of mine tried archiving two hundred thousand emails in one go, and it crashed his account so bad that he had no access for many days - and Google had to send an engineer in to solve this issue.

Even smaller batches (20k, 30k emails) would cause gmail to hang for some time - an hour, an evening - well, you get the picture. I have heard from more friends who recently tried to do something like this, and it worked quite well for them, thank you very much.

So now I'm gonna take a deep breath and attempt to archive 20,000 emails in one go. Wish me luck.

[cahuk]

Mail backups with OfflineImap

I have three mail accounts and I backup them all to my iMac with OfflineImap (installed through Homebrew). OfflineImap synchronizes emails between two repositories. That means that if I delete a message on my mail server the next time I run OfflineImap it will delete it on the local repository, too. But it will delete a message also on my server if I delete first a message on my local repository. That is a very powerful feature.

Imagine that you have a thousands of messages and want to delete the messages that are not important to you anymore. You could set up a script on your local machine that would filter and delete the meesages you don't need anymore (I store my mails in Maildir so every single message is a plain text file). The next time you run OfflineImap it will delete those localy deleted messages on your server, too.

Follow Steve Losh's fantastic tutorial how to set up OfflineImap. Please don't enter your mail password in plain text. Instead use that awesome Python script that invokes Keychain Access (only once; so make it sure that you click on Always allow).

When OfflineImap finishes his job, I run a rsync script that syncs the local mail repository with a repository on Dropbox.

[elisp-solves-problems]

Telling auth-source (and smtpmail-send-it) about my changed email password

I recently had to change my email password, as properly dictated by work's IT policies. I use offlineimap to get my mail, mu4e to read it, and smtpmail-send-it to send it.

(setq message-send-mail-function 'message-send-mail-with-sendmail message-send-mail-function 'smtpmail-send-it smtpmail-stream-type 'starttls smtpmail-default-smtp-server "pod51019.outlook.com" smtpmail-smtp-server "pod51019.outlook.com" smtpmail-smtp-user "[email protected]" smtpmail-smtp-service 587)

Offlineimap manages its password stuff from a config file at ~/.offlineimaprc. I updated that file and I was able to continue retrieving my mail with no hiccoughs. I then updated my ~/.authinfo file as well with the new password, but I continued to get the authentication failed error.

Sending failed: 535 5.7.3 Authentication unsuccessful in response to base64-encoded-password

That was pretty odd, considering that I changed the source file and all. I tried digging through smtpmail.el a little bit, looking for occurences of password or auth or something. I happened to see a defun called auth-source-search which got me into auth-source.el. Doing an apropos-command for "auth" led me to auth-source-forget-all-cached.

Apparently the auth sources like ~/.authinfo are cached for a customizable length of time, so I just had to clear the cache and everything is back in running order!

[cahuk]

Offlineimap and some troubles

Before a few weeks offlineimap stopped to work.

I do not know what was the reason for that but after some time of research I found what was wrong.

To make it clear, I am using the "pythonfile=" option in my offlineimaprc config file that enables using the Keychain Access app in OS X because I do not store any plain text passwords in my config file so the python file which is defined with the option above redirects at every login the Keychain Access app where are stored my passwords.

The error I got every time the last few weeks was:

'NoneType' object has no attribute 'group'.

To make it short, I found out that somehow my login internet passwords from Keychain Access for offlineimap were deleted. How, I do not know buy luckily I found it out running offlineimap with the "-1 -d imap" option. I saw that the login failed and I started looking around what could go wrong and found what went wrong.

If you are using offlineimap for making backups of your email accounts, don't store any plain text passwords in your config file and use Keychain Access instead.

Here is a great tutorial by Steve Losh for the offlineimap configuration (also for the Keychain Access option).

UPDATE:

Here is a how-to if you want to use gpg instead of Keychain Access.

[marcorucci]

Backup gmail with offlineimap: folders and labels.

Here are a couple details about backing up a gmail account through the IMAP protocol using [offlineimap][offlineimap]. Gmail uses **IMAP folders**, that we can consider as normal filesystem folders, and also **labels**, often called tags in other applications. The problem is that the IMAP protocol (or at least Maildir) does not know anything about labels. So this is how gmail deals with this discrepancy: Gmail uses only 4 real IMAP folders\*: * `[Gmail]/All Mail` * `[Gmail]/Drafts` * `[Gmail]/Spam` * `[Gmail]/Trash` This means that every email in your account, apart from trashed ones, spam or drafts, are stored in the IMAP folder `[Gmail]/All Mail`. If we synchronize our Gmail account (e.g. with offlineimap), the Gmail server will transorm the labels as real IMAP folders. For example, synching with offlineimap without any `folderfilter`, we get the following directory structure: [Gmail] All Mail Drafts Important Sent Mail Spam Starred Trash INBOX Personal Receipts Travel Work Every labeled email (note that even `[Gmail]/Sent Mail` is a label), will be duplicated because any email will always be stored in `All Mail` and will also be stored in the folder(s) corresponding to its label(s). If you have a crappy internet connection, for backup purposes, we can fetch only the `[Gmail]/All Mail` folder. This can be achieved setting `folderfilter` in the `.offlineimaprc` file as: folderfilter = lambda foldername: foldername in ['[Gmail]/All Mail'] Note, however, that in this way we lose the label structure. If instead we want to use the downloaded Maildir for bidirectional synchronization, i.e. we want to use a local mail client like mutt, then we have to fetch everything, thus downloading and storing duplicated emails. When synching back, the Gmail IMAP server will know how to handle IMAP folders as labels, and will not duplicate anything server-side. A sample `offlineimaprc` file is: ## For a complete offlineimaprc example, see: https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf [general] ## Possible options: Quiet Basic TTYUI Blinkenlights ## NB: only TTYUI and Blinkenlights are capable of prompting for the password ui = TTYUI accounts = gmail [Account gmail] localrepository = gmail-local remoterepository = gmail-remote [Repository gmail-local] type = Maildir localfolders = ~/.gmail ## This will locally replicate the IMAP folder structure, instead of the default separator '.' that flattens it. sep = / [Repository gmail-remote] type = Gmail remoteuser = [email protected] # remotepass = ## One-way synching. Perfect for backups. readonly = True ssl = yes cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1 ## Sync only 'All Mail': we lose the folder/labels structure, but this is the only way to not duplicate emails. # folderfilter = lambda foldername: foldername in ['[Gmail]/All Mail'] More information at the following pages: http://askubuntu.com/questions/23287/what-can-i-use-to-automate-backups-of-gmail http://pbrisbin.com/posts/mutt_gmail_offlineimap https://github.com/OfflineIMAP/offlineimap/ http://support.google.com/mail/bin/answer.py?hl=en&answer=82367&topic=1668961&ctx=topic \* I did not find this explicitly anywhere but this is what I understood reading [this google IMAP support page][support]. [offlineimap]: http://support.google.com/mail/bin/answer.py?hl=en&answer=82367&topic=1668961&ctx=topic [support]: http://support.google.com/mail/bin/answer.py?hl=en&answer=82367&topic=1668961&ctx=topic

[cahuk]

Command line

I use the command line more and more.

I started backing up my email with offlineimap. Now I do backups of the email folders with rsync because offlineimap deletes all local emails I delete on the remote. Have to figure out how to avoid that.

I installed mu today for indexing my maildirs.

Homebrew has it.

Install it with "sudo brew install mu".