Entity Linking with Wikipedia and Wikidata on spaCy v3.7

[RistovaIvona]

I'm planning to perform custom training with spaCy for entity linking using Wikidata. Could you provide guidance on creating a KnowledgeBase, especially since the latest spaCy version needs InMemoryLookupKB? I'm also considering refactoring the code in the current GitHub repository (https://github.com/explosion/projects/tree/master/nel-wikipedia) to match the latest version. Do you think this is a good approach, or is there a simpler way? Any helpful tips or tricks would be welcome.

[rmitsch]

Hi @RistovaIvona! InMemoryLookupKB is identical to the previous KnowledgeBase class - the renaming was done as part of a refactoring that will make knowledge bases more customizable and capable in spaCy v4.

Having said that, some resources that might be useful for you:

• wikid is a spaCy project for pulling Wikidata dumps, preprocessing and storing them in a SQLite database.
• The EL benchmark project leverages wikid to use Wikidata as knowledge base. The dataset used for training and evaluation is Mewsli-9, but it should be easy enough to adjust the scaffolding to train with any other dataset.

In general I'd recommend adjusting the linked EL benchmark project - I think you'll have less work updating that one to your requirements than the (considerably older) nel-wikipedia project.

If you have any specific questions, feel free to drop them here.

[rmitsch]

To be clear, did you run all of the following steps?

• wikid_download_assets
• wikid_parse
• wikid_create_kb

[RistovaIvona]

Yes, for both questions.

[rmitsch]

What's the content of wikid/assets and how large are the files in there, respectively?

[RistovaIvona]

wikidata_entity_dump.json.bz2 - 13.3GB
wikidata_entity_dump_filtered.json.bz2 - 98.3MB
wikipedia_dump_filtered.xml.bz2 - 112 MB
en-wikipedia_dump.xml.bz2 - 258MB

[RistovaIvona]

Should I set 'False' in the filter field?

[rmitsch]

wikidata_entity_dump.json.bz2 - 13.3GB
wikidata_entity_dump_filtered.json.bz2 - 98.3MB
wikipedia_dump_filtered.xml.bz2 - 112 MB
en-wikipedia_dump.xml.bz2 - 258MB

Both the Wikipedia dump as well as the Wikidata dump are considerably too small for them to be the full dumps (they should be around 21 and 80 GB, respectively). Try downloading them directly:

• English Wikipedia dump
• Wikidata dump

Should I set 'False' in the filter field?

Yes, that's important for the processing steps. It doesn't explain why you ended up with only a subset of the data though. So as next steps: download the dumps directly and set filter to "False".

[RistovaIvona]

I went through the entire process again, setting the filter to 'False' and using the --extraoption, but without success. (I have again this as output: start: 3, end: 4, surface_form: McCarthy, label: PERSON, id: NIL)
Currently, I am downloading both the Wikipedia dump and the Wikidata dump datasets, and I will upload them manually. I should then be able to continue the process from the wikid_parse stage?

[RistovaIvona]

Hello, I am facing an issue because the database is extremely large, and I'm running into out-of-memory problems, even when using powerful machines. Now, I want to create and use a filtered version of the database. By default, the existing filtered version contains only articles and entities mentioning "New York" or "Boston." I would like to use entities wherever people are mentioned (entity_id = "Q5", property_id = "P31"). How can I achieve this (either through code or by creating a new database on my own)?

[rmitsch]

You can have a look at scripts/wiki/demo_dump.py - this goes through the dump, keeps only entities/articles containing the terms in configs/filter_terms.txt, and stores two new dumps (one Wikidata and one Wikipedia dump). There is currently no support to filter by exact entity or property IDs though.

Wheree are you running into OOM problems? We had a similar complaint very recently, so we might look into fixing some of these memory issues.

[RistovaIvona]

I was at a point in the project where I need to create a kb ('spacy project run wikid_create_kb'). I was using a virtual machine with the following specifications: Machine type: e2-highmem-16, vCPUs: 16, Memory (GB): 128, Max number of persistent disks (PDs): 128, Max total PD size (TiB): 257, and Maximum egress bandwidth (Gbps): 16, along with a Boot Disk of 500GB.

[rmitsch]

Hm, 128 GB memory should be enough - we didn't use more either. Anyway, we'll mitigate the memory bottlenecks in the code. It should be possible to use generators at least partially to reduce memory requirements. I can't tell you when we'll get around to doing this (I hope in the next two weeks, but I can't promise that).

If you want to tackle this yourself meanwhile, I recommend looking into this thread which links to the relevant places in the code.

[lashmore]

Hello @rmitsch and @RistovaIvona , I'm following this thread and I'm currently on the wikid_parse step, and it's been running for hours now. I'm currently on iteration ~30,000,000. Wondering how long this parse takes? They didn't add tqdm so I'm not sure how far along I am. Here's what I see currently:

(nel) $ weasel run wikid_parse

================================ wikid_parse ================================
Running command: spacy project run parse wikid --vars.language en --vars.filter False

=================================== parse ===================================
Running command: env PYTHONPATH=scripts python ./scripts/parse.py en False
Parsing entity data: 30864146it [6:01:14, 677.90it/s] 

[lashmore]

So to update any interested parties:

For the wikidata parse, it looks like the total # entities was ~107,000,000.

I didn't set gpu_id to 0 before I ran the parse code, but looking at the code, I don't see any code that would benefit from a gpu to begin with - I think it's only for the train step. What a waste that it's running on a GPU VM though! Wish someone would have warned me to run these initial steps on a CPU machine!

[geekyayush]

I feel you man

[lashmore]

Well I ran through training but I'm not really happy with the results. It looks like training didn't really learn much, and it only performed 1000 steps.

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57                                                                              
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        81.36        79.43        83.38    0.57                                                                              
  0     800          49.23    90.69    92.83    88.65    0.00    0.00    0.00        81.07        79.15        83.08    0.57                                                                              
  0    1000          50.56    90.69    92.83    88.65    0.00    0.00    0.00        81.33        79.41        83.35    0.57   

I set max_training_steps to 0 and max_epochs is also 0 (which means unlimited for both), so the patience parameter now controls early stopping, but we'll see if that helps or is just a continued trend of the above for even longer training times...

[svlandeg]

@lashmore : hm, that certainly doesn't look like the model is really training/learning. I doubt increasing patience will help anything.

Can you share your config just for my information? And how many examples are there in training and in dev?

[lashmore]

I agree @svlandeg! It's been running for longer and the lack of learning continues!

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57                                                                              
  0     600          51.67    90.69    92.83    88.65    0.00    0.00    0.00        81.34        79.42        83.36    0.57                                                                              
  0     800          49.24    90.69    92.83    88.65    0.00    0.00    0.00        81.06        79.14        83.07    0.57                                                                              
  0    1000          50.56    90.69    92.83    88.65    0.00    0.00    0.00        81.33        79.41        83.35    0.57                                                                              
  0    1200          48.11    90.69    92.83    88.65    0.00    0.00    0.00        81.54        79.61        83.56    0.57                                                                              
  0    1400          53.94    90.69    92.83    88.65    0.00    0.00    0.00        81.03        79.11        83.04    0.57                                                                              
  0    1600          48.79    90.69    92.83    88.65    0.00    0.00    0.00        81.95        80.02        83.99    0.57                                                                              
  0    1800          47.96    90.69    92.83    88.65    0.00    0.00    0.00        81.21        79.29        83.23    0.57                                                                              
  0    2000          51.66    90.69    92.83    88.65    0.00    0.00    0.00        80.98        79.06        82.99    0.57                                                                              
  0    2200          49.63    90.69    92.83    88.65    0.00    0.00    0.00        80.94        79.03        82.95    0.57                                                                              
  0    2400          51.35    90.69    92.83    88.65    0.00    0.00    0.00        81.28        79.36        83.30    0.57                                                                              
  0    2600          48.78    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57                                                                              
  0    2800          48.35    90.69    92.83    88.65    0.00    0.00    0.00        81.89        79.95        83.92    0.57                                                                              
  0    3000          50.02    90.69    92.83    88.65    0.00    0.00    0.00        81.76        79.83        83.79    0.57                                                                              
  0    3200          46.28    90.69    92.83    88.65    0.00    0.00    0.00        81.54        79.61        83.56    0.57   

Here's how training is called:

#!/bin/bash

gpu_id="${6:--1}"

# Runs training. Expects as arguments:
#   (1) dataset ID,
#   (2) run name,
#   (3) language,
#   (4) config file name,
#   (5) max. steps.
#   (6) GPU information if GPU is to be used.
PYTHONPATH=scripts python -m spacy train configs/$4 \
          --paths.dataset_name $1 \
          --output training/$1/$2 \
          --paths.train corpora/$1/train.spacy \
          --paths.dev corpora/$1/dev.spacy \
          --paths.kb wikid/output/$3/kb \
          --paths.base_nlp wikid/output/$3/nlp \
          --paths.language $3 \
          --training.max_steps $5 \
          -c scripts/custom_functions.py \
          --gpu-id $gpu_id

And the bash script is called like:

bash scripts/train.sh "mewsli_9" "cg-default" "en" "nel.cfg" 0 0

Here is my complete config:

[paths]                                                                                                                                                                                                   
dataset_name = null                                                                                                                                                                                       
language = null                                                                                                                                                                                           
train = ""                                                                                                                                                                                                
dev = ""                                                                                                                                                                                                  
raw = null                                                                                                                                                                                                
init_tok2vec = null                                                                                                                                                                                       
kb = ""                                                                                                                                                                                                   
base_nlp = ""                                                                                                                                                                                             
vectors = "${paths.base_nlp}"                                                                                                                                                                             
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[system]                                                                                                                                                                                                  
seed = 342                                                                                                                                                                                                
gpu_allocator = null                                                                                                                                                                                      
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[nlp]                                                                                                                                                                                                     
lang = "en"                                                                                                                                                                                               
pipeline = ["senter","ner","entity_linker"]                                                                                                                                                               
disabled = []                                                                                                                                                                                             
before_creation = null                                                                                                                                                                                    
after_creation = null                                                                                                                                                                                     
after_pipeline_creation = null                                                                                                                                                                            
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}                                                                                                                                                          
                                                                                                                                                                                                          
                                                                                                                                                                                                          
[components]                                                                                                                                                                                              
                                                                                                                                                                                                          
[components.senter]                                                                                                                                                                                       
source = "${paths.base_nlp}" 

[components.ner]                                                                                                                                                                                          
source = "${paths.base_nlp}"                                                                                                                                                                              
component = "ner"

[components.entity_linker]
factory = "entity_linker"
entity_vector_length = 64
incl_context = true
incl_prior = true
labels_discard = []
get_candidates = {"@misc":"spacy.CandidateGenerator.v1"}

[components.entity_linker.model]
@architectures = "spacy.EntityLinker.v1"
nO = null

[components.entity_linker.model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v1"
pretrained_vectors = null
width = 96
depth = 2
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true


[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null

[initialize.components]

[initialize.components.entity_linker]

[initialize.components.entity_linker.kb_loader]
@misc = "spacy.KBFromFile.v1"
kb_path = ${paths.kb}

[initialize.tokenizer]

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}


[training]
train_corpus = corpora.train
dev_corpus = corpora.dev
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.2
patience = 10000
eval_frequency = 200
accumulate_gradient = 2
max_epochs = 0
max_steps = 500
annotating_components = ["senter"]
frozen_components = ["senter","ner"]
before_to_disk = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
nel_micro_p = 0.0
nel_micro_r = 0.0
nel_micro_f = 1.0


[pretraining]


[optimizer]
@optimizers = "Adam.v1"
learn_rate = 0.001
beta1 = 0.9
beta2 = 0.999
L2 = 0.0
eps = 0.00000001
grad_clip = 1.0
L2_is_weight_decay = true
use_averages = true

And here is the info on my corpora:

import spacy
nlp = spacy.load("en_core_web_lg")
from spacy.training import Corpus
corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
len([el for el in train_data])    # Output: 6336

corpus = Corpus("corpora/mewsli_9/dev.spacy")
dev_data = corpus(nlp)
len([el for el in dev_data])    # Output: 2536

corpus = Corpus("corpora/mewsli_9/dev.spacy")
test_data = corpus(nlp)
len([el for el in test_data])    # Output: 3804

I used the pipeline as specified here, but for the download_mewsli pipe I had to do some sleuthing about how to map the SVN revision number to a Git commit in the repo it was downloading files from. I believe I did it correctly / that the corpora were created as desired. And according to Table 1 here, the doc counts match up if you sum all 3 data partitions, you'll get the # docs reported for English, which this pipeline seems only to extract English data (I could tweak the download_mewsli9 pipe to be multilingual data instead but I didn't). What do you see?

[svlandeg]

What confuses me, is that you have nel_micro_f = 1.0 in your weights config, but yet the score from the original output log doesn't correspond 1-1 with the NEL_MICRO_F column. That's a detail though, and not related to the EL training (or absence of it).

That said, the F-score is actually pretty high (>80%) and the loss really low, making me wonder whether the EL ML model actually has a job left to do. What I mean by that is this: the EL component has two parts: 1) candidate generation and 2) ML-based reranker. Part 1 is a dictionary-based approach with the KB, it's only Part 2 that is actually learned. Imagine you having a dataset where each Named Entity would only have 1 candidate ID, then in fact the reranker has no job left to do, and whatever similarity-based ranking it attempts to learn, wouldn't make a difference.

This requires looking more into the data: what kind of entities are in your .spacy files? What IDs are in the gold data? How many candidates would each entity get from the KB? (just a quick manual inspection of a handful of examples from your training data will give you some idea).

[lashmore]

@svlandeg - so thankful to be talking to someone "in the know"! I admit, I'm still learning a lot of the "guts" of some spacy classes, like Corpora and InMemoryLookupKB, etc, but here's what I attempted to do to answer your questions about the Examples in the training data and how many candidates are produced from the KB

Here's the first Example from the train.spacy Corpus, which I believe is a generator object housing Examples.

import spacy
nlp = spacy.load("en_core_web_lg")  # the base model NEL was trained with

from spacy.kb import InMemoryLookupKB
from spacy.training import Corpus

corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
ex = next(iter(train_data))
print(ex.reference)   # Outputs: "Two men fined over 2006 German train crash that killed 23. Two men have been fined for their failure to prevent the 2006 Lathen maglev train accident in Germany..." 
print(ex.reference.ents)                                                                                # Outputs:   (Transrapid, Osnabrück)
print([cat for cat in ex.get_aligned_ner() if cat!="O"])                              # Outputs:   ['U-NIL', 'U-NIL']
print([tok.ent_kb_id_ for tok in ex.reference if tok.ent_kb_id_ != ""])     # Outputs:   ['Q157018', 'Q2916']
print([tok.ent_kb_id_ for tok in ex.predicted if tok.ent_kb_id_ != ""])     # Outputs:   []   <- Sanity check, as training hasn't started

kb = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=64)
kb.from_disk("wikid/output/en/kb/")

# The first entity generates only 1 candidate from the KB, as you guessed 
cand = kb.get_candidates(ex.reference[89:90])  # Corresponds to "Transrapid" entity Q157018 in reference doc
print(len(cand))            # Outputs: 1
print(cand[0].entity)    # Outputs: 15928567685345592367
print(cand[0].entity_)  # Outputs: `Q157018`

# The second entity generates 11 candidates from the KB
cand = kb.get_candidates(ex.reference[115:116])  # Corresponds to "Osnabrück" entity Q2916 in reference doc
print(len(cand))            # Outputs: 11
print(cand[0].entity)    # Outputs: 14312696856104742181
print(cand[0].entity_)  # Outputs: `Q2916`
print(cand[1].entity_)  # Outputs: 'Q5940'

Is it possible that the InMemoryLookupKB entity vectors are just THAT good already? How does the KB compute the vectors behind the scenes as it was parsing Wikipedia/Wikidata?

[svlandeg]

That's a great start to dig more into the data! Just as a small comment, you don't have to loop through your tokens in ex.reference, you could just run through ex.reference.ents and for each entity in it (which will be of type Span) print the kb_id - that's slightly cleaner than printing tok.ent_kb_id_ especially as one span can consist of multiple tokens.

Now that you have this code, you could probably have it run on your entire dataset, fetch for each entity the list of candidates like you're doing here, and count the len of those list, to obtain some sort of frequency list: how many entities have one candidate? how many have 11? etc. Would be interesting to better understand the challenge at hand.

Is it possible that the InMemoryLookupKB entity vectors are just THAT good already?

Here's another experiment I would run if I were you to get to the bottom of this. Go back to your config file which reads

[components.entity_linker]
...
incl_context = true
incl_prior = true

and run two more training steps.
Once with

incl_context = false
incl_prior = true

This won't learn anything, this setting basically disables the ML component (step 2 I wrote about before) - it will just take the prior probabilities of the candidates. Then we'll see whether this in fact already obtains 80% F-score by itself.

And then:

incl_context = true
incl_prior = false

This ONLY uses the similarity ML component to rerank in step 2 -> it won't use the prior probability. I'd be interested to see whether the learning is still "flat" when setting this. If it is, it might still be because there's usually only 1 candidate, or something else is going on still.

How does the KB compute the vectors behind the scenes as it was parsing Wikipedia/Wikidata?

If I remember correctly, the vectors represent the description we get from Wikidata, e.g. "district in Lower Saxony, Germany" for https://www.wikidata.org/wiki/Q5940 and "German city" for https://www.wikidata.org/wiki/Q2916. These descriptions are encoded using the base vector model, and the similarity component then compares that vector to the sentence in which the entity is used. Usually, the Wikidata descriptions are pretty concise and informative, and if you use e.g. en_core_web_lg as vector model, that should already be pretty capable of encoding the signal of these descriptions.

[lashmore]

Great ideas for debugging this @svlandeg!

Here's the distribution of candidate counts across the training Corpus's entities:

import spacy
from spacy.kb import InMemoryLookupKB
from spacy.training import Corpus
from tqdm import tqdm 

nlp = spacy.load("en_core_web_lg")
corpus = Corpus("corpora/mewsli_9/train.spacy")
train_data = corpus(nlp)
kb = InMemoryLookupKB(vocab=nlp.vocab, entity_vector_length=64)
kb.from_disk("wikid/output/en/kb/")

it = iter(train_data)
cands_lens = []
for ex in tqdm(it):
    for ent in ex.reference.ents:
        cand = kb.get_candidates(ent)
        cands_lens.append(len(cand))

from collections import Counter
dist_of_lens = Counter(cands_lens)
del dist_of_lens[0]    # delete counts associated with 0 candidates
print(dist_of_lens)

Results in:

Counter({1: 8991,
         2: 2299,
         3: 1227,
         4: 762,
         5: 544,
         6: 399,
         8: 318,
         7: 315,
         9: 306,
         10: 253,
         12: 191,
         13: 168,
         14: 168,
         16: 144,
         19: 134,
         11: 132,
         21: 130,
         15: 116,
         26: 115,
         17: 115,
         20: 113,
         22: 107,
         18: 95,
         1664: 88,
         56: 88,
         28: 85,
         24: 85,
         23: 79, .... })

So 8991 (39.3% of entities) of them only have 1 candidate, but a total of 13858 (60.6%) entities have >1 candidate. This doesn't account for possible duplicate entities in the training corpus, but I would have had to do NEL to write that code to begin with, so this is the best snapshot I can think of ha ha.

Ran with incl_context = false and incl_prior = true and got:

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        86.62        84.57        88.77    0.59
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        86.67        84.62        88.82    0.59

Ran with incl_context = true and incl_prior = false and got the below. I'm surprised it's not learning much either!

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
  0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        46.81        45.70        47.97    0.45
  0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        48.18        47.05        49.38    0.46
  0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        47.99        46.86        49.18    0.46
  0     600          51.68    90.69    92.83    88.65    0.00    0.00    0.00        48.16        47.02        49.35    0.46
  0     800          49.23    90.69    92.83    88.65    0.00    0.00    0.00        47.89        46.75        49.07    0.46
  0    1000          50.54    90.69    92.83    88.65    0.00    0.00    0.00        48.28        47.13        49.47    0.46
  0    1200          48.09    90.69    92.83    88.65    0.00    0.00    0.00        48.06        46.92        49.25    0.46
  0    1400          53.94    90.69    92.83    88.65    0.00    0.00    0.00        48.38        47.24        49.58    0.46
  0    1600          48.81    90.69    92.83    88.65    0.00    0.00    0.00        49.26        48.10        50.49    0.46
  0    1800          47.86    90.69    92.83    88.65    0.00    0.00    0.00        49.19        48.02        50.41    0.46
  0    2000          51.51    90.69    92.83    88.65    0.00    0.00    0.00        48.65        47.50        49.86    0.46

Another thing I noticed is that this benchmarking NEL code was written 2 years ago (commit was dated July 4, 2022), so at best they were developing in Spacy v3.3.1, and I also see in the requirements for the entire repo that spacy should be spacy>=3.0.0,<3.6.0. I'd previous been running training with Spacy v. 3.6.1 in a Conda environment with Python v. 3.10. I wondered if something happened to the NEL training code after 3.6.0 that's causing the above behavior, so I tested Spacy v 3.5.4 and got the same training behavior for the first three evals. i.e., it doesn't appear to be a package version issue:

E    #       LOSS ENTIT...  SENTS_F  SENTS_P  SENTS_R  ENTS_F  ENTS_P  ENTS_R  NEL_MICRO_F  NEL_MICRO_R  NEL_MICRO_P  SCORE 
---  ------  -------------  -------  -------  -------  ------  ------  ------  -----------  -----------  -----------  ------
0       0           1.08    90.69    92.83    88.65    0.00    0.00    0.00        83.73        81.75        85.81    0.58
0     200          62.05    90.69    92.83    88.65    0.00    0.00    0.00        81.27        79.34        83.28    0.57
0     400          50.64    90.69    92.83    88.65    0.00    0.00    0.00        81.17        79.25        83.19    0.57

[svlandeg]

Huh. So basically using no context makes your F-score better, almost 87%, which feels ridiculously high to me, especially considering the fact that you have 60% of entities with more than 1 candidate. It might just be that the prior probabilities are that good. BUT still something is definitely off as we don't see any learning for incl_context = true and incl_prior = false either.

We'll have to dive into this in more detail, I want to run some experiments myself to better understand what may be going on. Is there any chance you can share your .spacy files with me? I'll definitely keep them confidential and for debugging only.

[lashmore]

I can 100% send them to you because the data came from Google research! I just sent you email with the train/dev/test partitions. Very thankful for your eyes! I also sent you the conda env I'm working in, just in case you see anything missing in there, too. Thanks for your input @svlandeg !

[svlandeg]

Ok, looking into this more. We should look into the NER component. What is it doing? From the config, it's sourced from base_nlp, which is the vectors_model from wikid, which is probably en_core_web_lg. But is en_core_web_lg's NER component actually able to identify the entities from Mewsli-9? Because if not, there will be virtually no entities for the EL to train on.

We shouldn't ignore the fact that the ENTS_F column reads 0%, that seems to imply that the entities aren't getting matched.

[Luisgarchi]

Hello maintainers and contributors,

I am following this forum to implement the nel benchmark and got the following error on the wikid_parse step.

Looking at the error and examining the code in __init__.py, wikidata.py and wikipedia.py within the wikid/scripts/wiki directory it appears to me, to have successfully finished executing all the steps (given that the function read_texts is the last step, which finishes executing as per the print statement "Processed 23644888 articles" shown in the terminal before the error).

The size of my sqlite DB stands at 20.2 GB which seems to correspond to another person in this forum but not the 12GB mentioned by rmitsch. I also ran some sanity check queries on the main tables of the db and everything seems to be in order.

Machine specs: RAM 32GB, CPUs 16, Storage 500 GB (wikid_parse took ~ 24 hours)

• Is it safe to proceed ignoring this error?
• What can we attribute this error to? (My guess is python / weasel is trying to check the integrity of some file using md5 but it is to big to load into memory, since I dont know the path I assume it is something "big" like either the db or the dumps).
• Can I expect to run into memory issues in the create_kb step if using only 32GB.

Persisting alias-entity prior probabilities: 100%|█| 28351143/28351143 [
Parsing article texts: 6717273it [4:11:21, 445.39it/s] 
Processed 23644888 articles.
  Of which viable (with article ID and text): 6717273 (28.41%)
    Of which processed (title in entity table): 6717273 (100.00%)
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/spacy/cli/_util.py", line 87, in setup_cli
    command(prog_name=COMMAND)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/core.py", line 783, in main
    return _main(
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/core.py", line 225, in _main
    rv = self.invoke(ctx)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 42, in project_run_cli
    project_run(
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 113, in project_run
    update_lockfile(current_dir, cmd)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 270, in update_lockfile
    data[command["name"]] = get_lock_entry(project_dir, command)
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 287, in get_lock_entry
    outs = get_fileinfo(project_dir, command.get("outputs", []))
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/cli/run.py", line 308, in get_fileinfo
    md5 = get_checksum(file_path) if file_path.exists() else None
  File "/home/luisgarchi/.local/lib/python3.10/site-packages/weasel/util/hashing.py", line 33, in get_checksum
    return hashlib.md5(Path(path).read_bytes()).hexdigest()
  File "/usr/lib/python3.10/pathlib.py", line 1127, in read_bytes
    return f.read()
MemoryError```