Use jinja for chat templates

[nsarrazin]

We can use @huggingface/jinja for parsing chat templates and fetch chat templates from the hub directly. This would allow for a single source of truth for model templates.

[julien-c]

(cc @xenova for viz)

[xenova]

And luckily, we already have @xenova/transformers as a dependency (which adds tokenizer-specific functionality to @huggingface/jinja), so we can upgrade to >=2.14, and then use it as follows:

import { AutoTokenizer } from "@xenova/transformers";

// Load tokenizer from the Hugging Face Hub
const tokenizer = await AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1");

// Define chat messages
const chat = [
	{ role: "user", content: "Hello, how are you?" },
	{ role: "assistant", content: "I'm doing great. How can I help you today?" },
	{ role: "user", content: "I'd like to show off how chat templating works!" },
];

const text = tokenizer.apply_chat_template(chat, { tokenize: false });
// "<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"

const input_ids = tokenizer.apply_chat_template(chat, { tokenize: true, return_tensor: false });
// [1, 733, 16289, 28793, 22557, 28725, 910, 460, 368, 28804, 733, 28748, 16289, 28793, 28737, 28742, 28719, 2548, 1598, 28723, 1602, 541, 315, 1316, 368, 3154, 28804, 2, 28705, 733, 16289, 28793, 315, 28742, 28715, 737, 298, 1347, 805, 910, 10706, 5752, 1077, 3791, 28808, 733, 28748, 16289, 28793]

[mishig25]

Thanks to #850, there is a new field in model info called tokenizer which can be used for calling

const tokenizer = await AutoTokenizer.from_pretrained(model.tokenizer);

which is already the case here

chat-ui/src/lib/components/TokensCounter.svelte

Lines 12 to 15 in df27d77

	async function getTokenizer(_modelTokenizer: Exclude<Model["tokenizer"], undefined>) {
	if (typeof _modelTokenizer === "string") {
	// return auto tokenizer
	return await AutoTokenizer.from_pretrained(_modelTokenizer);

[nsarrazin]

Closing this as the feature is now released 😄