On: Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

[hedgehog]

128GB on a Ryzen 395 is enough but not by a lot. Around 90GB for weights plus 10GB to 20GB for KV cache and checkpoints. I don't know about 64GB of VRAM for less than around $2500 by itself.

[bitexploder → hedgehog]

Tesla V100? Would have to be a 3-ish bit quant in 64GB ram