Amazed at the duplicate filtering algorithm

[alicangok]

Hady, I'm amazed that you implemented a small algorithm to filter out highlights from similar "locations" within a book. Thanks for this

(Today, I was looking at my My Clippings.txt manually, and I realized that Kindle's handling of it is basically "write only". When you make a very little change to a highlight, let's say you expand it by a couple of words, it creates a completely new entry in My Clippings.txt. But somehow, this plugin is able to filter out those possibly "duplicate" entries, and provide a cleaner output. It essentially cleans out My Clippings.txt which the amazon programmers have failed to do.)

Unfortunately I know nothing about TypeScript at alI. If you have the time, could you point me towards which file you achieve this filtering, I'm curious.

[hadynz]

@alicangok Sorry for taking such a long time to respond to this. I remember reading this on the fly but never found your message again. I just discovered it now being in the Discussions section.

I've moved all parsing and processing of My Clipping highlights to the following library https://github.com/hadynz/kindleClippings.

If you are interested in the algorithm that I used, this is the crux of it. I do the following:

1. Read all my clipping blocks in order
2. Reverse all blocks (to go through them in reverse)
3. Dedupe blocks by using the following criteria - book title, highlight type (e.g. bookmark, note, highlight), location, content