safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge #5
Description
I tried to train the EasyRec model via python create_roberta_small.py, I downloaded the dataset and the contents of the baseline_embedders directory before running the command. However, I encountered the following error:
/media/***/sur/paper/EasyRec/baseline_embedders/roberta-base
Traceback (most recent call last):
File "/media/***/sur/paper/EasyRec/create_roberta_small.py", line 19, in <module>
base_model = AutoModel.from_pretrained("./baseline_embedders/roberta-base").cuda()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/***/miniconda3/envs/sur-easyrec/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/***/miniconda3/envs/sur-easyrec/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3456, in from_pretrained
with safe_open(resolved_archive_file, framework="pt") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
Here is my conda environment information:
# packages in environment at /media/***/miniconda3/envs/sur-easyrec:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 0.28.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2024.9.24 h06a4308_0
certifi 2024.8.30 pypi_0 pypi
charset-normalizer 3.4.0 pypi_0 pypi
filelock 3.16.1 pypi_0 pypi
fsspec 2024.10.0 pypi_0 pypi
huggingface-hub 0.26.2 pypi_0 pypi
idna 3.10 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
markupsafe 3.0.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.4.2 pypi_0 pypi
numpy 1.25.2 pypi_0 pypi
nvidia-cublas-cu11 11.11.3.6 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.8.87 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.8.89 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.8.89 pypi_0 pypi
nvidia-cudnn-cu11 8.7.0.84 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.3.0.86 pypi_0 pypi
nvidia-cusolver-cu11 11.4.1.48 pypi_0 pypi
nvidia-cusparse-cu11 11.7.5.86 pypi_0 pypi
nvidia-nccl-cu11 2.19.3 pypi_0 pypi
nvidia-nvtx-cu11 11.8.86 pypi_0 pypi
openssl 3.0.15 h5eee18b_0
packaging 24.2 pypi_0 pypi
pillow 11.0.0 pypi_0 pypi
pip 24.2 pypi_0 pypi
psutil 6.1.0 pypi_0 pypi
python 3.11.10 he870216_0
pyyaml 6.0.2 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
safetensors 0.4.5 pypi_0 pypi
scikit-learn 1.5.2 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
sentence-transformers 3.0.1 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setproctitle 1.3.4 pypi_0 pypi
setuptools 75.1.0 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
sympy 1.13.3 pypi_0 pypi
threadpoolctl 3.5.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
tokenizers 0.19.1 pypi_0 pypi
torch 2.2.1+cu118 pypi_0 pypi
torchaudio 2.2.1 pypi_0 pypi
torchvision 0.17.1 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
transformers 4.40.0 pypi_0 pypi
triton 2.2.0 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024b h04d1e81_0
urllib3 2.2.3 pypi_0 pypi
wheel 0.44.0 pypi_0 pypi
xz 5.4.6 h5eee18b_1
zlib 1.2.13 h5eee18b_1
I think this is caused by a module version problem, but I don't know how to solve it, can you help me?