New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does AutoAWQ support multi-threading CPU? #597

Open

sdecoder opened this issue Aug 23, 2024 · 0 comments

sdecoder commented Aug 23, 2024

Greetings everyone.

Server configuration: a modern CPU with multiple cores/large memory/a relatively weak GPU with insufficient VRAM(16G);
In such case, it is impossible to use the GPU to quantize the model due to low VRAM;
So is it possible to leverage the multiple cores of CPU?
I have tried to set the device_map="cpu", the work has been bound to CPU, but with ONLY ONE core. So it runs really slowly.

Can anyone provide any idea/hint to solve this issue?
Thank you all :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment