llmc is an off-the-shell tool designed for compressing LLM, leveraging state-of-the-art compression algorithms to enhance efficiency and reduce model size without compromising performance.
English doc is here.
Chinese doc is here.
Community:
- Sep 9, 2024: 🔥 We fix exporting quantized LLM to vLLM(see here). Moreover, we provide some configs of our best practice towards superior performance (see Best Practice here).
-
Sep 3, 2024: 🚀 We support opencompass to eval llmc model. Follow this doc and have a try!
-
Aug 22, 2024: 🔥We support lots of small language models, including current SOTA SmolLM(see Supported Model List). Additionally, we also support down stream task evaluation through our modified lm-evaluation-harness 🤗. Specifically, people can first employ
save_transmode(seesavepart in Configuration) to save a weight modified model. After obtaining the transformed model, they can directly evaluate the quantized model referring to run_lm_eval.sh. More details can be found in here. -
Jul 23, 2024: 🍺🍺🍺 We release a brand new version benchmark paper:
LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit.
Ruihao Gong*, Yang Yong*, Shiqiao Gu*, Yushi Huang*, Chengtao Lv, Yunchen Zhang, Xianglong Liu📧, Dacheng Tao
(* denotes equal contribution, 📧 denotes corresponding author.)
Instead of focusing on the best practice, We modularly and fairly benchmark LLM quantization considering calibration data, algorithms, and data formats. With detailed observation and analysis, we provide various types of novel points for performance and method improvements under different configurations. With the powerful toolkit LLMC and comprehensive insights, future LLM researchers can efficiently integrate suitable algorithms and low-bit formats for their applications, thereby democratizing the compression of large language models.
-
Jul 16, 2024: 🔥We support Wanda/Naive(Magnitude) for llm sparsification and layer-wise mix bits quantization now!
-
Jul 14, 2024: 🔥We support rotation based quantization QuaRot now!
-
Jul 4, 2024: 📱 We open our discussion channel. If you have any questions, please join our community:
-
May 17, 2024: 🚀 We support some advanced large models, e.g., LLaVA, Mixtral, LLaMA V3 and Qwen V2 now. Have a try!
-
May 13, 2024: 🍺🍺🍺 We release our quantization benchmark paper:
Ruihao Gong*, Yang Yong*, Shiqiao Gu*, Yushi Huang*, Yunchen Zhang, Xianglong Liu📧, Dacheng Tao
(* denotes equal contribution, 📧 denotes corresponding author.)
We modularly and fairly benchmark the quantization techniques considering calibration cost, inference efficiency, and quantized accuracy. Near 600 experiments on diverse models and datasets provide three insightful takeaways on the calibration data, algorithm pipeline, and quantization configuration selection. Based on the takeaways, a best practice for the LLM PTQ pipeline is designed, to achieve the best accuracy and efficiency performance balance under various scenarios.
-
Mar 7, 2024: 🚀 We release the quantization part of a powerful and efficient LLM compression tool. Notably, our benchmark paper is coming soon😊.
- Quantize LLMs, e.g., Llama2-70B, OPT-175B, and evaluate their PPL on only one A100/H100/H800 GPU💥.
- SOTA compression algorithms align with the origin repos, for users to choose from, and users can sequentially employ multiple algorithms on one LLM💥.
- Transformed model (
save_transmode inquantpart in Configuration) exported by our tool with a specifical compression algorithm can go through naive quantization by multiple backends, e.g., Lightllm, TensorRT-LLM to get a specifical-compression-algorithm-optimized model, which the corresponding backend can infer 💥. - Our compressed model (
save_lightllmmode inquantpart in Configuration) with a shallow memory footprint can be directly inferred by Lightllm💥.
-
Clone this repository and install packages:
# install packages cd llmc pip install -r requirements.txt
-
Prepare models and data.
# After downloading LLMs from huggingface, prepare calibration and evaluation data as follows: cd tools python download_calib_dataset.py --save_path [calib data path] python download_eval_dataset.py --save_path [eval data path]
-
Choose an algorithm to quantize your model:
# Here's an example about Awq: cd scripts # Modify the path of llmc, ``llmc_path``, in the bash file. You can also choose one config # placed in ``llmc/configs/quantization/Awq/`` to quantize your model, or your own # config referring to those we provide by changing the ``--config`` argument in run_awq_llama.sh. bash run_awq_llama.sh
To help users design their configs, we now explain some universal configurations in all configs we provide under llmc/configs/:
-
model:model: # Replace by the name of the class in ``llmc/models/*.py``. type: Llama # Replace by the path of your model. path: model path torch_dtype: auto
-
calib:# Note: some algorithms do not need ``calib``, like naive... So, you can remove this part. calib: # Replace by the calibration data name, e.g., pileval, c4, wikitext2, or ptb, downloaded before. name: pileval download: False # Replace by the path of one of the calibration data, e.g., pileval, c4, wikitext2, or ptb, # downloaded before. path: calib data path n_samples: 128 bs: -1 seq_len: 512 # Replace by the function name in ``llmc/data/dataset/specified_preproc.py``. preproc: general seed: *seed
-
eval:# If you want to evaluate PPL of your pretrained/transformed/fake_quant model. eval: # You can evaluate the pretrain, transformed, fake_quant model, and set the position # you want to evaluate. eval_pos: [pretrain, transformed, fake_quant] # Replace by the name of the eval data, e.g., c4, wikitext2, ptb or [c4, wikitext2], # downloaded before. name: wikitext2 download: False path: eval data path # For 70B model eval, bs can be set to 20, and inference_per_block can be set to True. # For 7B / 13B model eval, bs can be set to 1, and inference_per_block can be set to False. bs: 1 inference_per_block: False seq_len: 2048
-
save:save: # ``save_trans`` is True, which means you want to export the transformed model, e.g., parameter-modified # model, whose performance and structure are the same as the original model, and users can # utilize naive quantization to the transformed model to obtain the same performance as # the specifical-algorithm-quantized model. save_trans: False # ``save_lightllm`` or ``save_trtllm`` is True, which means you want to export a real quant model, e.g., # low-bit weights with weight and activation quantization parameters. save_lightllm: False # ``save_fake`` is True means you want to export fake_quant model, e.g., # dequantized weight with activation quantization parameters. save_fake: False save_path: ./save
-
quant:quant: # Replace by the class name in ``llmc/compression/quantization/*.py`` method: OmniQuant # weight-only quantization does not have ``act`` part. weight: bit: 8 symmetric: True # Quantization granularity: per_channel, per_tensor, per_head (not recommended). granularity: per_channel group_size: -1 # Calibration algorithms: learnble, mse, and minmax (default). calib_algo: learnable # Utilize Stright-Through Estimation, which is necessary for learnable # calibration algorithms. ste: True act: bit: 8 symmetric: True # Quantization granularity: per_token, per_tensor granularity: per_token ste: True # Static quantization (quantization during calibration)or dynamic # quantization (quantization during inference). static: True # This part is designed for specific algorithms, users can refer to # those we provide to design their own. special: let: True lwc_lr: 0.01 let_lr: 0.005 use_shift: False alpha: 0.5 deactive_amp: True epochs: 20 wd: 0 # If quant_out is True, employ the outputs of the former quantized block as the # calibration data of the proceeding block. quant_out: True
✅ BLOOM
✅ LLaMA
✅ LLaMA V2
✅ OPT
✅ Falcon
✅ Mistral
✅ LLaMA V3
✅ Mixtral
✅ Qwen V2
✅ LLaVA
✅ StableLM
✅ Gemma2
✅ Phi2
✅ Phi 1.5
✅ MiniCPM
✅ SmolLM
You can add your own model type referring to files under llmc/models/*.py.
✅ Naive
✅ AWQ
✅ GPTQ
✅ OS+
✅ AdaDim
✅ QUIK
✅ SpQR
✅ DGQ
✅ OWQ
✅ HQQ
✅ QuaRot
✅ Naive(Magnitude)
✅ Wanda
✅ ShortGPT
We develop our code referring to the following repos:
- https://github.com/mit-han-lab/llm-awq
- https://github.com/mit-han-lab/smoothquant
- https://github.com/OpenGVLab/OmniQuant
- https://github.com/IST-DASLab/gptq
- https://github.com/ModelTC/Outlier_Suppression_Plus
- https://github.com/IST-DASLab/QUIK
- https://github.com/Vahe1994/SpQR
- https://github.com/ilur98/DGQ
- https://github.com/xvyaward/owq
- https://github.com/TimDettmers/bitsandbytes
- https://github.com/mobiusml/hqq
- https://github.com/spcl/QuaRot
- https://github.com/locuslab/wanda
- https://github.com/EleutherAI/lm-evaluation-harness
If you find our LLM-QBench paper/llmc toolkit useful or relevant to your research, please kindly cite our paper:
@misc{llmc,
author = {llmc contributors},
title = {llmc: Towards Accurate and Efficient LLM Compression},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/ModelTC/llmc}},
}
@misc{gong2024llmqbench,
title={LLM-QBench: A Benchmark Towards the Best Practice for Post-training Quantization of Large Language Models},
author={Ruihao Gong and Yang Yong and Shiqiao Gu and Yushi Huang and Yunchen Zhang and Xianglong Liu and Dacheng Tao},
year={2024},
eprint={2405.06001},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{gong2024llmcbenchmarkinglargelanguage,
title={LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit},
author={Ruihao Gong and Yang Yong and Shiqiao Gu and Yushi Huang and Chentao Lv and Yunchen Zhang and Xianglong Liu and Dacheng Tao},
year={2024},
eprint={2405.06001},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2405.06001},
}