Correctly checking if token is in tokenizer

[VeritasJoker]

I was running encoding and found that the gpt2 base_df has some tokens that are marked as False in the in_gpt2-xl column so I looked into it. Turns out there are some sort of weird things going on in the tokenizer.

As seen by the example below, the word You're is tokenized into 'ĠYou' and "'re", and both tokens should exist in the gpt2-xl tokenizer. However, when we tokenize "'re" again in tfsemb_LMBase.py, it is further tokenized into "Ġ'" and 're', thus causing the helper function to return False here when it should have been True:

247-pickling/scripts/tfsemb_LMBase.py

Line 39 in 961fbec

if len(tokenizer.tokenize(x)) == 1:

Really not sure what's happening here. Is it because of the left padding or the double quotes vs single quotes? I will do some testing but if @hvgazula and @zkokaja have some ideas let me know! Thanks!

[zkokaja]

Why and where do we "tokenize "'re" again" ? Should we be using another column instead?

Good catch

[zkokaja]

option 1: remove in_ models and align on word_idx instead
option 2: look at using token instead of token2word?

[zkokaja]

waiting on #153 to include word_idx and this can be handled in encoding. and option 1 is robust to changes in tokenization strategies