Regarding llama3-70b-instruct

[chintanshrinath]

Dear
I am trying to load full model on A100-80 GB of 8 cores using below command.
docker run --gpus all --shm-size 1g -e HUGGING_FACE_HUB_TOKEN=$token -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-input-length 8000 --max-total-tokens 8010

However, it is not using all GPU core.
I also looked num_shard, but didn't get it.

Can you help here to to use all core and optimize the above command. The main concern is that we need to decrease inference time for production grade.
Thanks

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.