7B

[Soulscb]

No description provided.

[Soulscb]

Traceback (most recent call last):
File "/data/11103440/code_gen_eval_bin/awq_quantize.py", line 26, in
model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/awq/models/base.py", line 231, in quantize
self.quantizer.quantize()
File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 166, in quantize
scales_list = [
File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 167, in
self._search_best_scale(self.modules[i], **layer)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 330, in _search_best_scale
best_scales = self._compute_best_scale(
File "/usr/local/lib/python3.10/dist-packages/awq/quantize/quantizer.py", line 409, in _compute_best_scale
raise Exception
Exception

主要配置如下 quant_config = { "zero_point": True, "q_group_size": 32, "w_bit": 4, "version": "GEMM" }
量化到最后出错

[jacob-hansen]

+1, I have the same error with Gemma 2 27B-it (on AutoAWQ Gemma branch)

[Soulscb]

+1, I have the same error with Gemma 2 27B-it (on AutoAWQ Gemma branch)

do you solve it?

[raghavgarg97]

getting the same error,the issue is happens at 16th layer of gemma-2-27b-it as the input to it is a tensor of nans so the loss never decreases in finding the best scale
havent figured out the soln yet