How can I use Llama.cpp quantized models?

[NPap0]

Am I correct in saying that if I want to use the Llama.cpp models I would have to get the alpaca weights after getting access to them from META (Not from the notorious magnet link) and then run them through quantization via https://github.com/ggerganov/llama.cpp/issues and then get the output of that and use it in privateGPT or am I missing something? And is it a grey area to be distributing the final output(quantized models) because the weights are not 'free' for commercial use?

[andrewginns]

If you look around you can find that others have posted the complete quantised ggml format models including alpaca online.

[NPap0]

If you look around you can find that others have posted the complete quantised ggml format models including alpaca online.

Thought so but couldn't find anything and thought because the quantization format (llama.cpp) was changed recently there wouldn't be any such cases.
I'll continue looking around, thanks!

[thekit]

I solved getting some q5 models to work in this discussion here: #233