Skip to content

reduce memory usage during repack #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 7, 2025
Merged

reduce memory usage during repack #168

merged 1 commit into from
Mar 7, 2025

Conversation

ReinForce-II
Copy link
Contributor

solve #166

@wejoncy
Copy link
Owner

wejoncy commented Mar 7, 2025

Thanks for your contribution.

One question: Do you verify it of reducing memory usage? Could you please paste the number here?

Much appreciate.

@ReinForce-II
Copy link
Contributor Author 

python -m qllm --load Qwen/Qwen2.5-14B-Instruct-AWQ --eval --save /tmp/tmp0012 --pack_mode=GPTQ

w/o this change, peak memory usage is 66.8GiB.
w/ this change, peak memory usage is 23.0GiB.

I didn't do more tests.

@wejoncy wejoncy merged commit df20c15 into wejoncy:main Mar 7, 2025
2 checks passed
@wejoncy wejoncy linked an issue Mar 7, 2025 that may be closed by this pull request
Memory consumption about convert model format #166
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Memory consumption about convert model format
2 participants
@ReinForce-II @wejoncy