Agreed that glm-5.3-flash is a strong model. On a single 6000 pro you can (barely) fit a q2 quant in vram. I haven't used it extensively, but my initial feeling is that the q2 is quite a bit worse than q4 for this model. To run it comfortably at q4 with reasonable context length you really need 2 6000s.
When the qwen 4 series is released I am hopeful there'll be a strong model with the same architecutre. as qwen3.8-flash-next.
| 0 N/A N/A 321801 C llama-server 16540MiB
| 1 N/A N/A 321801 C llama-server 22404MiB
| 2 N/A N/A 321801 C llama-server 23480MiB
| 4 N/A N/A 321801 C llama-server 43982MiB
| 5 N/A N/A 321801 C llama-server 41656MiB
| 6 N/A N/A 321801 C llama-server 22366MiB
| 7 N/A N/A 321801 C llama-server 16208MiB
That's 3.86 bits per word I could run the 4 bpw one as well (there is still one spare gpu and another 27G on the ones listed above. DS4 uses about the same memory, is a little bit faster (though I suspect that by the time GLM 5.3 support is a bit more mature the speed difference will have evaporated).
Pretty much. And, interestingly, even though it is a little bit slower than most other models I can run here it is quite a bit more efficient with the number of tokens used so on longer runs it often comes out ahead in terms of wall clock time. YMMV of course this is for pretty specific workloads and there is a good chance that on other workloads it would perform worse. But so far it has been a pretty good model for us.
When the qwen 4 series is released I am hopeful there'll be a strong model with the same architecutre. as qwen3.8-flash-next.