It uses all if my 6 rtx3090s but runs 7 sessions at combined 200tok/s and prefill of 7k on a cheap pc (no threadripper, all gpus via usb4).
DeepSeek v4.1 Flash 1017 is better but it is also 3x the size... And I find running quant below 3.8bits unusable.
It uses all if my 6 rtx3090s but runs 7 sessions at combined 200tok/s and prefill of 7k on a cheap pc (no threadripper, all gpus via usb4).
DeepSeek v4.1 Flash 1017 is better but it is also 3x the size... And I find running quant below 3.8bits unusable.