On: Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU

[iv42]

If you've got RTX 5090, maybe try ninfer (https://github.com/Neroued/ninfer). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).

[Foobar8568 → iv42]

Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures.

https://github.com/Neroued/ninfer/blob/master/docs/performan...

Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract

And on my "own" "quick" benchmark, it's slower than vllm.

[iv42 → Foobar8568]

I don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...