Experiment with implementing AWQ for BERT models

[casper-hansen]

If we can speed up the BERT model, we will significantly increase the throughput of many cases. Experiment with SentenceTransformers first.

[z3ugma]

Would love this for image captioning with quantized speedup.
The kosmos-2 model from Microsoft would be another good candidate

[michaelfeil]

hej @casper-hansen I would be curious to implement this for https://github.com/michaelfeil/infinity. Do you see any road-blockers in regards to encoder only architectures? Will the GEMM kernels work for non-causal masked LMs?

[casper-hansen]

@michaelfeil AWQ/GEMM kernels can work for any linear layer. However, there is a challenge in applying it to BERT models because it lacks some scaling methods. For example, we would usually scale from a layernorm to a linear layer.

See more about the scaling of layers here:
https://github.com/casper-hansen/AutoAWQ/blob/main/awq/quantize/scale.py

I also created a PR for better scaling for Mixtral, which may be interesting to you:
#301