On: QuIP#: 2-bit Quantization for LLMs

[tarruda]

If this quantization method works with smaller models, it would enable running up to 33B models with only 12GB VRAM.

Especially important for democratizing access to Mistral MoE new model.

[MrNeon → tarruda]

IIRC quantizing small models causes a higher relative drop in the metrics.