dealing with message duplicates?

[rkiddy]

Long time listener, first time caller. (Well, not really but...)

I am hoping, really hoping, that mox will help me fix my duplicate emails problem. Is there any way it can do this? I have been very unhappily stuck in an outlook server for a long time and am looking forward to being out of it.

If I grab my emails off of the server, or import from a local disk, is there then some way that I can eliminate duplicates? Trying to move things around with an outlook server has left me with many, many dupes.

I would appreciate any suggestions.

[mjl-]

Hi @rkiddy! Mox does not deduplicate messages, essentially because it means accepting a message (on import or delivery) and then discarding it. It doesn't feel right, and users could/should normally expect each mail to be visible after an import/delivery. Mox does match up duplicate messages in a thread, and shows them as duplicate in its webmail interface. But I can understand it's annoying if all your messages have multiple copies.

Wouldn't it make more sense to just deduplicate the messages once? Either with a tool that speaks IMAP that connects to your IMAP account, fetches all the raw messages, calculates a cryptographic hash, identifies duplicates, and removes all but one copy of the duplicates? Or perhaps only based on the Message-Id header (though that may delete too many messages). Or alternatively, export all messages to Maildirs, and run a generic file deduplication tool on the Maildirs, and importing those Maildirs in mox.

I did a quick search and found https://github.com/quentinsf/IMAPdedup and https://github.com/kdeldycke/mail-deduplicate and https://github.com/adrianlopezroche/fdupes. I have not used any of these tools! If you're going to try any of these tools, I'm interested in hearing how it worked out!

[rkiddy]

Well. I got a "here let me search that for you"... I have been trying to find these kinds of tools but was not searching using the correct terms, or searching for the specific term. I have been looking for command-line access to my mail server and had not found anything. What search did you do? If it was not just "deduplicate my email".

And I will report on the usefulness of the tools.

[mjl-]

I think I looked for "email message deduplication", and some variants with IMAP mixed in, and things like "generic unix file deduplication". Will be interesting to hear how it works out!