More than five years ago, I decided that I wanted a terminal based e-mail management solution for the half a dozen e-mail addresses I read on any given day. My requirements were a unified inbox, tagging, and offline synchronization using imap. After testing a variety of solutions, I landed on a combination that satisfied my requirements, except for deleting e-mail from Gmail. Over the last two weeks, I finally found a solution that works reliably. In this post, I’ll describe my research, and how I designed a solution that met my needs.

Given that mutt is the industry standard for an open-source terminal based e-mail client, I knew that I wanted to use it as my point of user interaction with my e-mail. My second requirement was a unified inbox, parallel to my interaction in K-9 Mail preventing the need to navigate from inbox to inbox for new e-mail. After researching solutions, the one that was frequently returned in my search results was Notmuch. Notmuch is an incredibly fast, full text searchable system for e-mail. Its frontend is an API and a collection of CLI tools which absolutely exceeds my requirements. Once I decided that I was going to use notmuch, I looked for an integration with mutt, and discovered that it wasn’t as clean as I hoped. I researched further and discovered NeoMutt! The integration between the two was built in 2016 and includes virtual folders, which solves my unified inbox requirement. The last requirement was the ability to sync each complete e-mail account locally, do what I need to do with it, and then “push” the changes back up when I’m done. This makes e-mail management an activity that happens a couple of times a day, versus real-time (which is an incredible distraction).

I spent a considerable amount of time configuring each of the tools to work together, and generally, my workflow is as easy as the one I use on my mobile with K-9 Mail. On a regulas basis, I pull e-mail down from each of the accounts on my mobile (when on the move), review all of the new mail in a single unified inbox, and then mark for follow-up (and archive), archive, or delete, achieving “Inbox Zero”.

The issue I had with the solution was how Google implemented IMAP for their e-mail product. Through the support of various IMAP extensions, they were able to dynamically build IMAP folders from their special labels as well as labels that a user creates through the UI. For example, an excerpt of the system labels:

The challenge with this implementation is that any and all mail delivered to a given address is stored in All Mail unless explicitly expunged (a.k.a. deleted). Research by the offlineimap team determined that to delete an e-mail completely from Gmail, it needs to be “moved” to Trash and deleted from all other folders.

Were you able to figure the problem out?

If not, I’ll help.

Gmail only uses one folder, and adds labels to each piece of correspondence indicating its folder
Standard IMAP protocol only understands folders

Because of these two conditions, offlineimap pulls the same message for every folder that it exists in and saves it as a seperate file.

Note that the same piece of correspondence has two records; one in inbox, and one in archive. If I run a diff on both files, the output is the following:

The only difference is that the entry in the ‘All Mail’ folder has an \Inbox keyword, compared to the record in the Inbox folder having no labels at all. This now became a relatively simple search and replace problem that could be solved on the command line with one of my favorite tools, sed. I put together a list of steps that the script needed to take.

# Steps required to properly delete e-mail via IMAP (not JMAP) from GMail
# supported accounts, which also includes accounts hosted through Google
# Workspace
# 
# Assumptions, using offlineimap v8.0.0 with imaplib2 v3.0.6 and notmuch v0.37
# E-mails to be deleted are tagged with 'trash'
# 1. Find e-mail files that are not currently in trash and are tagged trash
# 2. Delete any records of each that are in folders other than archive
# 3. Remove all X-Keywords from records that are in archive
# 4. Remove MAILDIR hash from filename
# 5. Move the result to the corresponding trash folder in the correct account

Once I had the steps down, it became a development effort. A couple of loops, variables, and a lovely set of Shell Parameter Expansion techniques later, (kudos to @hsanson with their comment in a neomutt thread discussing a similar issue that they had solved), I built out the following:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Tidy to Trash
for i in $(notmuch search --output=messages not path:/trash/ and tag:trash)
  do 
    for j in $(notmuch search --output=files tag:trash and $i)
      do 
        ACCOUNT=$(echo $j | awk -F/ '{print $5}')
        DIR=$(echo $j | awk -F/ '{print $6}')
        if [ $DIR != 'archive' ]
        then 
          /usr/bin/systemd-cat echo "Remove $j in $DIR"
          rm $j
        elif [ $DIR == 'archive' ]
        then
          /usr/bin/systemd-cat echo "Remove X-Keywords from $j"
          sed -in 's/X-Keywords.*/X-Keywords:/g' $j
          FILE=${j##*/}
          FILE_NOUID=${FILE/,U=[0-9]*:/:}
          FILE_NOUID=${FILE_NOUID/,U=0-9]*/}
          /usr/bin/systemd-cat mv -uv $j ~/mail/$ACCOUNT/trash/cur/$FILE_NOUID
        fi
    done
done

I had used simpler expansions in the past on other scripts, but the set that was built out in lines 16 through 18 used regex to solve an issue that would be much harder otherwise. Per the implementation by offlineimap3, the maildir name is hashed using MD5 and appended to the filename of each piece of mail. During my testing, when I was moving files locally, and syncing the changes with the remote, I found duplicate mails and other inconsistencies. I realized after reviewing this piece of code that the mismatch of the hash to the changed folder was causing the issue. By removing the hash as part of the tidy operation, this ensured consistency between the local and remote mail repositories.

The entire script is in a gist for those that are interested and I’ll continue to update it if I encounter edge cases that need to be dealt with. I reused the trash loop to create an archive loop so that I can use a different notmuch tag (‘archive’) to handle moving mails from inbox to archive when in my unified inbox virtual folder.

I use three macros in my neomutt configuration for when I’m in a virtual folder.

macro index,pager \ca "<modify-labels>+archive\n<next-entry>" "+archive"
macro index,pager \cd "<modify-labels>+trash\n<next-entry>" "+trash"
macro index,pager \cu "<modify-labels>-trash\n" "-trash"

I won’t comment on how many hours I spent researching and eventually building this solution; it was a lot. I learned a lot about IMAP, and in general, how my mail tools could be configured to work together. I give a lot of credit to each of the development teams that maintain the tools that give me the opportunity to manage my e-mail in a way that’s convenient for me.

Related Posts