Detecting possible citekey collision via scripted import (debug-bridge)

[atensity]

Hi!

Thank you so much for the extension, it has been incredibly useful and an almost daily necessity. I did have a question though, about the plausibilaty of the following:

I am controlling the addition of entries to Zotero via the Debug Bridge plugin, since I want to be able to do add a new entry to Zotero from another program (Emacs) when given a URL. So far, the small script that I created which sends some javascript code to Zotero's (Standalone) instance for adding (via Zotero's translators/processors) has been working decently well and haven't really been running into any issues.

The current issue arises as I am trying to extend this a bit more. Basically what I want to do, is that if an entry already exists for a given URL, then instead of making a new entry, it should allow me to add a note to that entry in Zotero (i.e. if I want to make multiple notes for a given entry); or alternatively in a separates note file (which gets updated when I add things to Zotero through Emacs).

The issue I have, is that I am unable to really check whether a given would-be entry already exists. I planned to do this initially through URL only, but when going through Zotero's translators, the fields might be quite different for i.e. journal articles, books and webpages.
Then I though of using the CiteKey, but for this I would need to:

1. Find out the citekey for the new would-be entry, after Zotero's translators
2. Look at all current existing citekeys (for a given target library/collection)
3. If entry exists, then just return that previous entry exists, with it's item ID, citekey etc
   1. What to do when this happens (i.e. adding a note to it) can be taken care of later
4. If it does not exist, make a new entry, and optionally add a note.

I run into the issue that when it tries to go through Zotero's translators, I'm guessing BBT already assigns a citekey with a postfix e.g. citeKey->citeKeya. This means that if I attempt to run a new URL through a translator, then try to check if would-be citekey that would match any existing ones, it will always return false, since BBT automatically gives it a unique citationkey. This also means that it will create a new entry when given the same URL

I know that Zotero's browser connector has the whole "Right Click on text selection > Add note to zotero", but this has the same issue, in that if I do this twice on the same webpage, it creates two different entries with the notes attached separately.

I did take a peek in how better-bibtex did it (i.e. KeyManager.propose in key-manager.ts; using the Formatter), but I've been unsuccessful in tinkering with it.

Basically to summarise, I want to be able to extract the would-be citekey and check for possible key collisions when given a URL (which will go through Zotero's translators).

Is there a better way to go about this, and am I overcomplicating things?

Note: This is just meant for a locally running instance of Zotero, not for any sharing between people or anything.
PS: Is there a recommended way for extending certain functionality of e.g BBT or Zotero other than making a plugin or via debug-bridge?

(also Happy Easter!)

[retorquere]

The issue I have, is that I am unable to really check whether a given would-be entry already exists. I planned to do this initially through URL only, but when going through Zotero's translators, the fields might be quite different for i.e. journal articles, books and webpages.

That is true, but since you're doing the item creation yourself, you could perhaps add the originating URL to the item in its extra field. Using that you could even prevent creating duplicates. If you're OK with first creating a duplicate, another option would be to uses Zotero's duplicate detection it uses in its merge facility.

Then I though of using the CiteKey, but for this I would need to:

1. Find out the citekey for the new would-be entry, after Zotero's translators

But after the translator is done, it is no longer a would-be entry, it would be an actual entry. It's not technically impossible to hook into Zotero's machinery and access the item to be saved and prevent it from actually being saved, but it would not be trivial.

I run into the issue that when it tries to go through Zotero's translators, I'm guessing BBT already assigns a citekey with a postfix e.g. citeKey->citeKeya. This means that if I attempt to run a new URL through a translator, then try to check if would-be citekey that would match any existing ones, it will always return false, since BBT automatically gives it a unique citationkey. This also means that it will create a new entry when given the same URL

That's not exactly right. BBT does not know how items get into your library, be it import, typing in, or syncing. After an item is created by any of these ways, and at every modification after, BBT will regenerate the citation key. So BBT has not already assigned a citekey when item is imported, but it will do so very shortly after creation (by default, 5 milliseconds after create/change).

During import, Zotero assembles the new item and saves it. BBTs Zotero.BetterBibTeX.KeyManager.propose will work with the unsaved item if you want it, but to get the unsaved item you'd have to monkey-patch the translator infrastructure.

I know that Zotero's browser connector has the whole "Right Click on text selection > Add note to zotero", but this has the same issue, in that if I do this twice on the same webpage, it creates two different entries with the notes attached separately.

Yes. This is a longstanding issue in Zotero that people have asked about for years and years. It's not an easy problem to solve though.

I did take a peek in how better-bibtex did it (i.e. KeyManager.propose in key-manager.ts; using the Formatter), but I've been unsuccessful in tinkering with it.

The formatter isn't exposed but I wouldn't mind exposing it. Propose indeed adds the disambiguator so that will never find duplicates, but you could probably just check whether proposal.slice(0, -1) exists. Even getting the formatted key without the disambiguator would not be foolproof, because if you had Einstein1905 and Einstein1905a, you wouldn't know which of the two is "On a Heuristic Point of View Concerning the Production and Transformation of Light" or "On the Electrodynamics of Moving Bodies". And if you would remove the first, the remaining would stay as "Einstein1905a". Duplicate detection just isn't simple.

Basically to summarise, I want to be able to extract the would-be citekey and check for possible key collisions when given a URL (which will go through Zotero's translators).

When you say "will go", that means you don't have any information on the item yet. The translator reads what's behind the URL and makes Zotero-information from it. Before it does that, all you have is an URL. BBT can't do anything with that.

Is there a better way to go about this, and am I overcomplicating things?

Not overcomplicating, dedup is just complicated.

PS: Is there a recommended way for extending certain functionality of e.g BBT or Zotero other than making a plugin or via debug-bridge?

Not really, no.