[Kernel] add bfloat16 support for gptq kernel

[jinzhen-lin]

Some models would overflow when using fp16 inference (e.g. Deepseek-V2), so we should add bfloat16 support for quantization kernel. This PR add bfloat16 support for gptq kernel.

Related issue: #2149

main changes:

• add bfloat16 input/output support for cuda kernels
• dequant qweight to bfloat16 in proper ways.

NOTE: Currently, bfloat16 kernel may be much slower than float16 on >=sm80,<sm90 device since the support for atomicAdd with bfloat16 is not native (see description of atomicAdd). Increase the value of BLOCK_KN_SIZE can much improve the performance, but I don't sure if this will affect other situations.

[mgoin]

@alexm-neuralmagic should we support bfloat16 here the same way as done for gptq_marlin?

[jinzhen-lin]

@alexm-neuralmagic should we support bfloat16 here the same way as done for gptq_marlin?

The bfloat16 gptq kernel have serious performance issue now, I may optimize it recently.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[mgoin]

This has been resolved long ago on SM >= 80 since the GPTQ Marlin kernels support bfloat16, closing as resolved

vllm/vllm/model_executor/layers/quantization/gptq_marlin.py

Lines 74 to 76 in 2adb440

	@classmethod
	def get_supported_act_dtypes(cls) -> List[torch.dtype]:
	return [torch.half, torch.bfloat16]