Count system prompt tokens

[mishig25]

On updating model settings:

On updating assitants settings:

[gary149]

There're some errors with some models I think 👀

[mishig25]

There're some errors with some models I think 👀

The reason being is that: https://huggingface.co/meta-llama/Llama-2-70b-chat-hf is a gated model. Therefore, the tokenizer code from frontend tries to grab the files but it is gated.

For now, the behaviour is that: if the model is not gated, the token count is shown. If it is gated, nothing appears.

Wwe have two poaaiblities:

1. Keep the current behaviour where no tokens are shown on gated models
2. Move the tokens counting code to backend/server so that tokens can be shown on all models.

wdyt ?

[gary149]

I think the current behavior is good. But it shouldn't even try to tokenize if the tokenizer is not available (here it throws 100s of console errors).

[mishig25]

I think the current behavior is good. But it shouldn't even try to tokenize if the tokenizer is not available (here it throws 100s of console errors).

handled it. Only the one error gets thrown in this case

[gary149]

Just don't do it on llama-2? (as its gated it will never work) maybe it could be in the config.

[nsarrazin]

When it works it seems to work well, but I sometimes get console errors when switching between different models in the settings without reloading the page? But if I reload the page the tokenization works again. Not sure what is causing it

[gary149]

When it works it seems to work well, but I sometimes get console errors when switching between different models in the settings without reloading the page? But if I reload the page the tokenization works again. Not sure what is causing it

Yes trying to reproduce that too 👀

[gary149]

This won't work on local models with chat-ui. I would probably set an explicit key in the config for models that are compatible.

[mishig25]

Updates:

Added model.tokenizer config. You can supply:

string - which would be used in transfomer.js AutoTokenizer

OR

{
    tokenizerUrl: string;
    tokenizerConfigUrl: string;
}  - which would be used to construct transfomer.js PreTrainedTokenizer

See .env.template file of this PR. Important: when testing locally make sure you have copied model.tokenizer from .env.template of this PR

Fixed reactivity issues mentioned above

here and here

Editing model system prompts

Screen.Recording.2024-03-19.at.12.21.00.PM.mov

Creating assistnat

Screen.Recording.2024-03-19.at.12.08.24.PM.mov

Editing assistnat

Screen.Recording.2024-03-19.at.12.20.18.PM.mov

[gary149]

use truncate value instead?

[mishig25]

handled in f47c6bb

[nsarrazin]

Works quite well! 🔥