How should resyncing work?

[hadynz]

In the interest of putting out a MVP that proves the value of this plugin, I implemented a simple and crude manner for resync. If a book has been resycned before, the plugin will never overwrite or modify this file even if new highlights have been detected.

This ensures that any changes to your files are always preserved.

But it does mean that you will not get any new highlights syncing... unless, you manually delete the file yourself and run the sync again.

This current behaviour is not necessarily intuitive for many users. So I am keen to get feedback on how you would expect resync to work.

[eleanorkonik]

The most intuitive & easy thing for me would be that resync overwrites no matter what, if the file is in the same folder it was in initially. But I use folders more than other people and tend to move files out of the "extractions" folder I use for zotero and kindle highlights/annotations extraction once I've finalized my formatting. Probably don't do this, heh.

The second option is to append the new "version" at the bottom with a header that says something like "added on x date" with bonus points if it detects which things are new highlights/annotations and appends only things that are different (such as if there was a change to the template).

But that's probably a lot more complicated.

Caveat: currently I'm paranoid and personally delete or rename and move any sync'd files I care about before a resync, and only edit the files after I've finalized reading it. The second option where I could trust the diffing would be really nice!

[pgomez]

First off, thanks for the work on this @hadynz and opening this discussion up.

The second option sounds amazing and would be super helpful. Maybe another option to add to this is to have a different code line in the template for updated highlights:

{% for highlight-u in highlights %}

This could allow someone to place any new highlights at the bottom of the page or above their notes. It provides extra flexibility, but keeping the same highlight structure/properties you have in place.

{% endfor %}

I may not have thought this out correctly, but this is what came to mind.

[nickmilo]

I actually prefer the current crude method fwiw

[eleanorkonik]

I definitely don't mind the current version! It's very straightforward really.

[davejunkie]

It would be nice if on re-sync if any new notes or highlights were found (maybe using the location) in the kindle book, it would then be added to the end of the MD note. If we are syncing then we have access to revisions. I do like what @eleanorkonik said about adding date added when the new notes or highlights are added.

[owenvachell]

Is it possible to add the books that have been synced to the plugin settings panel? Or maybe in a pop-up when the sidebar icon is clicked? Then include an option to re-sync an individual book, a list of books or all books? Perhaps also allow the user to specify if notes should be overwritten with new changes or appended to existing notes?

My personal opinion is that I'm fine with the crude method of overwriting the contents of existing notes. I'd imagine this would also be easier to implement and quicker to process as you wouldn't need to check if the highlight already exists in the note. I guess the simplest way would be @eleanorkonik suggestion to store the time of last sync and then append anything new since that time?

[hadynz]

That's a fair request. I'll be making a release soon that will allow you to visually see after every sync what every book that you've synced it, and what the status of the last sync session was for every book (e.g. synced, skipped, errored).

This is a first step to test out the visual paradigm. Can then add the ability for you to manually and on demand to request a resync for a specific file.

[doodoo787]

And maybe the option to "not sync" certain books ever? I have more than a dozen books that always want to sync but the notes are sooo old and useless I really don't want them in my vault).

[rbb428]

As for me, I think the current method is straight forward. When I finished a book, I sync all the note and highlights of that book. However if I am reading several books in the meantime, then I can't guarantee the plugins got all the highlights of the books which have been synced. So it may be confused sometime. Also if a e- book file in kindle is very large, like a series book (Harry Porter vol. 1-7), I may want to sync the highlights while I am still reading that book. Because I want to use obsidian to add more linked- note to the highlights. For example, I might highlight the description of a wand which I am interesting. And I will link a web- page Wand Woods
By J.K. Rowling to the note I synced through the plugin. So as for this point I pretty agree what @eleanorkonik has mentioned. The plugins can add a time-tag each time the book has been synced. Then it will have an option to sync the new change from kindle and add them at the bottom of the note.

[joelbeckham]

@hadynz, thank you for this plugin. It's already been so helpful to me!

While the crude method is already so useful to me, I would benefit from having an incremental resync. I often add new highlights and notes when I reread a book and it'd be nice to pull those in without losing the links and tags that I added to my existing imports.

I made a Kindle to Evernote importer a while back before I discovered Obsidian (Obsidian + your plugin is so much better!!) A few things I discovered:

• There wasn't a timestamp on highlights, so I couldn't use that to detect new highlights
• Occasionally 2 highlights near each other would get the same location, so that wasn't a reliable way to know if the highlight had already been imported.
• I ended up uniquely identifying the highlight with a format like "< location >.< highlight+note hash >" and that worked pretty well. If the highlight or note changed or was new, it would just get imported as a new item. I'm still getting used to Obsidian, but if you went that route, I wonder if putting something like that in a manual block id might work? Then you could compare it to the content in Obsidian (to detect if it had been modified, so you could know not to overwrite) and compare it to what's coming from Kindle to see if it's new.

One other idea I had that I couldn't use in Evernote, but might work here, is if a resync will change the existing file, you could instead do the resync to a different file next to the original, so if the original sync was "Escaping the Build Trap", the next iteration would be something like "Escaping the Build Trap (2021-05-28)". Then I could just use a diff tool to merge the differences how I like. That might not be a great option for non-devs though.

[hadynz]

From my highlights and that of others, I haven't seen a scenario were locations are duplicated.

That said. The most guaranteed way to identify a highlight is to hash the actually highlight content.

Now the difficulty is not here IMO. The challenge is where do you store such state information? Because a user is able to modify the template in any way that they want, I cannot make any assumptions that any block numbers or identifiers will be there.

So the options so far to keep track of state information for highlights that have been synced are:

A - keep track of state information as front matter metadata in a file. Risk exists that a user tampers with this metadata after it is written

B - keep track of sync data on a separate file on disk. I would need to configure the plug-in to listen to any file changes (e.g. rename, folder move) to know where the file is. I would not be able to detect changes that happen outside of obsidian.

So my preferred option so far is A

[joelbeckham]

Good points. I'm still getting used to the Obsidian way of doing things. That makes sense on the challenge of storing the state. Seems like option A would be better in this case.

Here's an example of where the location is duplicated:

[joelbeckham]

I probably have 3 or 4 duplicates like this in the 150 or so books I've imported.

[albionx]

This is a great plugin -- thanks for working on this.

In my case the current behavior is problematic, as I want to be able to write my own notes on Obsidian, referencing book highlights, before I finished reading the entire book. It would be much better for me if new highlights could be appended to the note.

I appreciate there is a problem with blocks that have been annotated by users. I suggest a two-fold solutions:

1. Provide a way for users to enable/disable the 'Re-sync existing books' option. Disabling it would be equivalent to current behavior.
2. Provide guidance to users who enable this setting, that their notes should be either in another 'note' (linking to it), or at least in another block (thus not breaking the hash de-duplication scheme). I'm not an experienced Obsidian user, but it seems like having this convention shouldn't be too problematic.

+1 to @owenvachell 's suggestion of having per-book sync'ing as well

[OogieM]

I've been playing with the plug-in and have run into many of the same issues as described here.

I really want a way to only sync books whose annotations have changes. Given the discussions above I really don't know how best to do it but here is my problem. I have about 1200 books in kindle format. I already have the issue that any books not bought through Amazon do not put any highlights into their cloud service so there is now way to sync them at all and it's nearly impossible to extract them into any usable format. Problem number 2 is that I do all my kindle reading on an iPad in the Kindle App so there is no clippings file so I can't extract out the clippings from any of those books. That already takes the kindle book population down to about 800 or so. But since the plug-in syncs all of them every time it's slow and I end up with hundreds of duplicates.

My use for the highlights is that I want them to come in with complete bibliographic information into my "inbox" folder initially. Then as I process them I make notes in between the highlight references, some were made as notes as I read the book and some are when I review the books. When I think I am done (at least for this iteration) I move the book note into the folder that contains my references and sources. So the initial location that is created upon sync is not the final location where the notes will live. I want all my annotations and quoted sections in a single note in Obsidian that I can link to. Plus I typically have about 10-15 books I am reading at any given time and if I need to get the most current highlights everything gets re-synced again.

When I re-read a book or go back to it to look up something I may make more highlights or notes or annotations on a book that has already been synced. But I don't want the new version to overwrite what I have in my vault that has been expanded to include my own thoughts.

I could live with a sync that assumes no personal notes or additions are being added within Obsidian but it's not my preference. I'd really prefer some sort of combination of a smart sync that will only add new things to an existing note if there is one and that won't re-sync if there are no changes but will catch entirely new books and also not overwrite anything I've added to my notes and handle new notes in one place and looking for existing notes in another.

To throw another wrink;le into the mess I'd also like a way to define what the note title is according to a template similar to the way I can define how the files are renamed in Zotero. In my case right now I'm using this as my template

Book-authorlastname-title with all spaces replaced by _ characters-kindle

I add the format of the book at the end because I am also collecting into Obsidian the notes and annotations that I have about paper books and the annotations I do when I am reading scientific papers that are in my Zotero system. Right now my book bibliography is in BookPedia. It can create a BibTex file but there is no way to automatically keep it updated that I can see. Im still playing with all of that. Ideally it would be similar to how Zotero works with Obsidian. The whole combining of my references and sources system in with Obsidaian notes on the various things I am researching or want torefer to is in flux but adding a way to get Amazon kindle highlights into Obsidian is a great addition.

[OogieM]

Yeah, I haven't thought of how to do state capture either. Then we're back to the user keeping all personal notes separate from the annotations. That is a limitation I can live with.

Is there any way to know from the Amazon side if any new annotations have been made? The equivalent of a date modified but on the Amazon servers? It won't be perfect, you still need to store somewhere the last date the plug-in looked at stuff but it might get part of the way there.

[hadynz]

Not easily.

If your highlights and notes are being imported from the My Clippings file on the Kindle, then there is a timestamp available for every interaction that you make. That makes identifying last modified date very easily.

However, the notes that get rendered through Amazon's notebook does not have any timestamps at all. So the only way to identify any new highlights is to keep track of state, do a dedupe, and reverse engineer what changes might have happened.

[OogieM]

The gotcha seems to be the applications vs a real kindle device. I've done some poking around in the database that stores the highlights for iOS but it's confusing because it also stores all the popular highlights for the book in question. OTOH it is an SQLite Datbaase and if you can manage to filter out the popular ones maybe ther eis a chance. I was poking at it but had to stop to work on my own programming projects and it was a couple of years ago. Not sure I can ressurect what I did but I'll look and see if I can find anything
that might be helpful.

[hadynz]

@OogieM This is so, so interesting and the first time that I heard about this. Are you saying that you've got a manual way to import highlights made on the iOS Kindle App by tapping into the SQLite store that it uses on device?

If there is a manual way, then there is a possibility that we can automate it.

If you can share with me the steps on how you do it manually, I can try it out at my end to validate that I can do it too (I have an iPhone). Once I do it manually, I can look at ways at automating it right from Obsidian.

[OogieM]

OK, I am actually in the middle of working in SQLite again and refreshing my knowledge so I'll see if I can find my notes (one reason I'm moving into Obsidian, so I can find that stuff) One fear I have is that the original database of annotations may now be obfuscated by either kindle app changes or iOS changes. I'll see what I can find this week.

[BricoleurSoul]

I would like to use this plugin as an integral part of my Obsidian PKM and research tool. The best way I can think of doing this is to be able to link individual word/phrases in the generated notes extracted from Kindle to MOC - or currently "under-construction evergreen notes." I am returning to review a book I last read about a dozen years ago due to a new piece I am currently reading (I had missed the importance of a piece in the first book and went back to highlight it in Kindle) but am unable to get that entry to sync with your plugin.

I do not want to delete the existing entry from your plugin as it would result in loss of all that work.

If the added functionality is possible by appending (better would be inserting just after the Kindle link line before the first entry in existing file) your plugin would revolutionize my ongoing refinement of my cybernetics research. Thanks for the work you've done so far !

ps. as a kluge I am going to try copying the existing generated entry from its existing location and then resyncing and since I still remember the few new entries I just made in Kindle I'll be able to highlight just those new ones in the new download at "cost" of having near duplicate entries for the book. Not pretty but workable in a pinch.

[kevinrothermel]

I wouldn’t be against the system creating a new note with a date stamp at the end of the file name that contained all of the highlights of the previous note along with any new ones. Users could then delete the older note. Seems like it might be a simple short term solution that would at least move things forward a bit by requiring less fiddling from users to get new highlights from a book into obsidian.

[OogieM]

That would be workable for me. I'd just need to know that so I will keep my own comments (except the actual notes taken attached to a highlight) out of the individual book notes. My only request would be that the date and timestamp can be adjustable as to format. If I were to add that to one of my books the note title/filename would look like this

Book-Swann-Three_Bags_Full-kindle_2021-07-09_13-47-49

That way I can parse out the type of source, a book, The author last name, the title, the format of the source and a date and time stamp for the file of the annotations associated with that item.

[edward-peters]

As a keen user of this fantastic plug in, I have been following this thread with great interest. For the plug in to work for me, it needs to be able to update highlights when I am reading a book and highlight further down in the text. I don't mind what solution is chosen, just so long as this option becomes available.

I am debating whether to pay for Readwise to achieve this, but would much prefer your plug in. Is there any hope of this being implemented in the near future?

Thanks again for this great plug in.

[hadynz]

I hope you held out from a Readwise subscription — if all that you wanted was access to your highlights. It grinds my gears that we pay for the Kindle device, and pay for our books, yet cannot access our own annotations.

Please give v1.0.x a spin and let me know if that has brought this issue of resyncing to bed.

Also have a read of my answer here for complete detail on how resyncing works as part of this latest change.

[edward-peters]

Thanks @hadynz for all your work on this. In the meantime I have taken out a Readwise subscription, as their new linked Reader application (in private beta) is a game changer for me, and makes the subscription worth it.

[Vicki-Davis]

I think it would be very helpful if you could select the book to import highlights from, then select which chapter to import. That way, if each chapter is treated separately, perhaps it would be able to consider a newly highlighted chapter as not previously synced so that a re-sync would pick up the new highlights.

I don't want to have to delete my previous notes in order to get new highlights and I often only use select chapters, then may go back and use other chapters later. I edit by chapter and I don't want my previous notes to be changed if I re-sync.

Thank you for considering my ideas.

[hadynz]

Thanks for the ideas @Vicki-Davis. Please try out the latest version of the plugin and let me know if how resyncing works for you or not.

It would be hard for me to implement your idea of considering highlights by "chapter" as that is information that is not known to the plugin or in your annotations.

[hadynz]

Thank you all for your patience on this issue being resolved.

3 weeks and 65 files later, I am happy to share with you that this feature has finally been implemented and shipped as of yesterday. Please try out v1.0.x and let me know what you think of how it works. It should work for anyone trying to sync from their My Clippings file or from Amazon cloud.

This solution works by introducing a breaking change to how files used to be synced before. To get resyncing working, I had to make two key changes:

Frontmatter

The plugin needs to keep track of "state" information after a sync takes place. To devise a portable solution that works on any machine you use Obsidian for, I opted to add syncing state as frontmatter to every book that you sync. It looks something like this:

---
kindle-sync:
  bookId: '37249'
  title: >-
    How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and
    Thinking – for Students, Academics and Nonfiction Book Writers
  author: Sönke Ahrens
  asin: B06WVYW33Y
  lastAnnotatedDate: 'July 13, 2019'
  bookImageUrl: 'https://m.media-amazon.com/images/I/81A0MWyFUeL._SY160.jpg'
  highlightsCount: 121
---

This state information, and primarily lastAnnotatedDate is used by the plugin to determine if there are any new highlights to be synced. I will also be using this information soon in another major release that introduces a user interface to see all your books and highlights visually, with a bunch of interesting features to interact with your data.

Block references

Running with @joelbeckham's idea, every highlight will always be prefixed with a block reference that contains a unique reference for a highlight by "hashing" its content. This was required so the plugin can (a) determine what highlights have not been synced yet and (b) where to insert new highlights in the file.

The combination of the above two changes allows the plugin now to intelligently figure out what highlights to sync and to also insert it in the right order in your file, irrelevant of any other changes that you've already made to the file. This should support workflows like progressive summarization were you need to bold and highlight annotations that you've made in file.

Breaking change: With all of this said, the above two changes does mean that I have introduced a breaking change that will require all users to resync all their annotations again as a one-off step.

Would love to hear your feedback and if this change has unlocked your workflows.

Cheers

[OogieM]

Testing this now. one minor quible. Can we change the format of the last annotated date to be a user preference? I always use dates as YYYY-MM-DD. Makes it a lot easier to search for dates and sorting if you use the date in filename works more accurately too.

[hadynz]

Storing the date by user preference (in the frontmatter) will introduce unnecessary complexity. But I think it makes sense for the default date to be in the YYYY-MM-DD format as it is much easier to parse than what I have used.

I can the last annotated date (formatted by user preference) to the # Metadata section. Would that be valuable in your opinion?

[hadynz]

@OogieM just implemented your feature request in 1.0.3. Last annotated date in the frontmatter is stored as YYYY-MM-DD format. You will see that if you sync a book with new annotations, or if you delete and re-sync again.

[OogieM]

Yes that would be helpful