On: QuIP#: 2-bit Quantization for LLMs

[skavi]

to clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit

[beacon294 → skavi]

It is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.

[skavi → beacon294]

So your experience isn’t representative of the work presented by this post? Or does llama.cpp use the same technique for quantization?

[beacon294 → skavi]

I don't know, hopefully it will help set general expectations.