-
Notifications
You must be signed in to change notification settings - Fork 204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi-Node Quantization using Ray? #601
Comments
Hi @paolovic, at the moment this is not something explicitly supported or even something that I have attempted. I suspect it could be possible, but it's not something that I have researched. If you do find the time, I would love to receive a PR with either code changes / docs on how to do this. |
Hoi @casper-hansen , |
Hi,
in theory I could get enough compute to host and quantize current models.
But it will be provided as multiple VMs, each with 2GPUs, each with 48GB VRAM.
Using these, I could create a Ray Cluster https://github.com/ray-project/ray with, e.g., 5 nodes, therefore in total 10 GPUs and 480 GB VRAM.
Is it possible to utilize this to quantize models with AutoAWQ?
Thank you very much!
Best regards
The text was updated successfully, but these errors were encountered: