### System Info

[Karliz24]

System Info

Hello developer,
The Llama-3 model was released today.

I want to convert this model to a hf model, but when I follow the readme, the following issue occurs.
File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in <module> main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in __init__ super().__init__( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__ slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in __init__ self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

I would really appreciate it if you could give me some guidance on how to solve this problem.
Please help me. thank you!!!

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

'python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py
--input_dir /workspace/00_develop/llama/llama3/llama3-model/Meta-Llama-3-8B
--model_size 7B
--output_dir /workspace/00_develop/llama/llama3/hf-llama3-model/hf_Meta_Llama_3_8B'

Error logs

raceback (most recent call last):
File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in
main()
File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main
write_model(
File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model
tokenizer = tokenizer_class(tokenizer_path)
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in init
super().init(
File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in init
slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in init
self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor
model = model_pb2.ModelProto.FromString(sp_model)
google.protobuf.message.DecodeError: Error parsing message

Expected behavior

no converting

Publicación original de @yuri-son en meta-llama/llama-recipes#445

[calmitchell617]

Copying and pasting my response to your original issue:

Hi, I am having this same issue. However, the code in question is in the Huggingface Transformers lib, not in llama-recipes. I've opened an issue there.