You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
l use the code below to add about 100 tokens to model and tokenizer,and the model doubles in size.
from transformers import AutoTokenizer,AutoModel
tokenizer = AutoTokenizer.from_pretrained("llama-7b-model")
model = AutoModel.from_pretrained("llama-7b-model")
characters = [
#100tokens
]
num_added_toks = tokenizer.add_tokens(characters)
tokenizer.save_pretrained("new-llama-7b-model")
model.save_pretrained("new-llama-7b-model")
In contrast, other language models such as BERT, BART, GPT-2, etc., that I have used previously do not exhibit such a significant increase in size.
I have also tried fine-tuning LLAMA1.3b, and after adding tokens, although the size nearly doubles, the entire fine-tuned model ends up slightly smaller than the original 1.3b model.
I am puzzled by this situation and unsure if it is normal. Do you have any insights into the reasons behind this?
Thank you.
The text was updated successfully, but these errors were encountered:
l use the code below to add about 100 tokens to model and tokenizer,and the model doubles in size.
In contrast, other language models such as BERT, BART, GPT-2, etc., that I have used previously do not exhibit such a significant increase in size.
I have also tried fine-tuning LLAMA1.3b, and after adding tokens, although the size nearly doubles, the entire fine-tuned model ends up slightly smaller than the original 1.3b model.
I am puzzled by this situation and unsure if it is normal. Do you have any insights into the reasons behind this?
Thank you.
The text was updated successfully, but these errors were encountered: