-
Notifications
You must be signed in to change notification settings - Fork 225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
数据在入过程中样本量减少 #682
Comments
XTuner 默认会拼接数据至 max length 以提升 gpu 利用率,导致实际 iter 和数据条数不符 https://xtuner.readthedocs.io/zh-cn/docs/acceleration/pack_to_max_length.html |
懂了,多谢解答 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
如图,原本有3w多样本, 最后就只有4k多,该如何定位该问题?
脚本如下:
rm -rf llama3_finetune_pth/*
output_dir=llama3_finetune_pth
config_py=xtuner/configs/llama/llama3_8b_instruct/llama3_8b_instruct_qlora_alpaca_e3.py
CUDA_VISIBLE_DEVICES=0,1 NPROC_PER_NODE=2 xtuner train ${config_py} --work-dir ${output_dir} --deepspeed deepspeed_zero2 --seed 1024
The text was updated successfully, but these errors were encountered: