[QA] about chart format，[UNUSED_TOKEN_146] means 146 token or self

[limoncc]

描述问题

about chart format，[UNUSED_TOKEN_146] means 146 token or self

tokenizer.decode([146])
print(a)
# <0x8F>
b = tokenizer.encode("[UNUSED_TOKEN_146]")
print(b)
# [1, 640, 2010, 15927, 8783, 3553, 304, 284, 10389, 332]

so what means?

[ZwwWayne]

[UNUSED_TOKEN_146] means the 146th unused token in pre-training, so it should be treated as a whole in tokenization. However, it seems that you are using a different tokenizer, because correct results should be:

In [1]: import torch
   ...: from transformers import AutoModelForCausalLM, AutoTokenizer
   ...: 
   ...: model_path = "internlm/internlm2-chat-7b"
   ...: 
   ...: tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)


In [2]: 

In [2]: tokenizer.encode('[UNUSED_TOKEN_146]')
Out[2]: [1, 92543]

In [3]: tokenizer.encode('[UNUSED_TOKEN_145]')
Out[3]: [1, 92542]

In [4]:

[limoncc]

It is appreciated for answering. because I use a gguf, i will check.

[ZwwWayne]

#619 provide more description.