How to load tokenizer config?

[linonetwo]

QwenLM/Qwen1.5#304 (comment)

They are also provided in tokenizer.json as the standard practice in transformers

Do we have an API to load this?

[giladgd]

There’s a JinjaChatWrapper you can use, and it’s automatically get used when the model includes a chat template.
You can run this command to chat with a model, and look for the chat wrapper metadata that’s printed before the chat starts:

npx -n node-llama-cpp chat <model file path>

You can also use fileInfo on a model instance to read metadata.

[linonetwo]

How about the tokenizer.json? I think it can solve wrong token in CJK language

[linonetwo]

I used to not passing the template, and still get bad result. So the problem must cause by tokenlizer, do you have guide to load it @giladgd ?

[giladgd]

You can access all the tokenizer related metadata on a model by using model.fileInfo.metadata.tokenizer.
The tokenizer.json you're talking about is probably the original file used in the HF format of the model.
model.fileInfo.metadata.tokenizer is a transformed version of that file adapted to llama.cpp and embedded into .gguf files

[linonetwo]

Thanks, so GGUF should have everything embedded. Then I have no idea why it performs so bad...