Find duplicated Documents

[patsch9]

Summary:
Filter documents with identic or nearly identic ocr content an list it.

Detail:
Create a task that compare the ocr content of the documents an list it when it matches a certain percentage. So i can find documents that i sometime have already insert an now uploaded or scanned it again. Than i can delete the duplicate.

[afiorillo]

I was going to suggest a similar feature as this, though a little more general -- content based deduplication. There are a few ways to do this:

• Using an image hash of the PDF content page for page
• Using a Jaccard-index or some other kind of fuzzy matching to detect suspected duplicates based on the OCR content

I will have some free time in February and may pick this up, as it would be really helpful. At least for me, it is not uncommon that I will find a document buried in a box and be unsure whether I've already scanned it.

[jurjevic]

Any update here?

[afiorillo]

Edit: moved this top-level reply to the thread.

None unfortunately. The free time in February vanished very quickly. I
think the way I would do this is similar to how the paperless_tesseract app
is designed, and just label any detections as suspected duplicates or
something of the sort. Ideally it would be a separate/optional plugin, but
I'm unsure how that would integrate.

[alexanderherfurtner]

Same from my side - it might be useful in some scenarios. At least if there is a heavy load to import to the DMS and to keep track of some mistakes during the process.

Is there any idea or plan to develop such a feature? Maybe I've some time in winter to do so.