Too bad you probably need something like an 8x B300 node to run this. Black Friday is coming soon though, but I fear this open intelligence will be pacing anyone's liquidity frontier so I'll probably keep trailing the frontier for a while with nvidia/qwen3.8-27B-NVFP4 on dual RTX PRO 4000 in TP=2. I can almost fit 450k context in there.
Models this size aren't targeting hobbyists running models in their bedrooms on gaming/surplus hardware bought with disposable income.
This model is for researchers who already have racks of equipment, inference providers who sell access to models, and for businesses who must (or choose to) use on-prem models. All of which have to pay to use the model if their revenue rises beyond certain thresholds: https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LI...
Wow... I'd like to know more. 40 analysts not spending 100USD/month is 4k/month saved. That means you break even in 27 months. But you probably used Claude much more than that.
Also what harness do you use? Any problems you encountered on the way?
The benefit is that you can use these models on a variety of different providers. You're not stuck with a single proprietary provider. Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.
I regularly use together.ai with models like this, where they're fully self hosting them and I can turn off all data retention, at a fraction of the cost of proprietary models. I recently discovered abliterated models (models edited to not refuse to work on security stuff) and hosts for them like refuseless.com, and that's an option for open source models, not for proprietary ones.
> Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.
Unfortunately due to their licensing requirements for modal-as-a-service providers, the pricing can’t go lower than Moonshot’s from what I can tell.
This is still great if like your example you trust together.ai more, or if different providers have better speeds/cache rates (though usually the original lab has the best cache hit rate).
And obviously if you’re a large enough service to afford to self host or finetune it then this is good.
I didn't dig into Kimi's MaaS legalese, but i dont really understand how that knee-high fence is supposed to work long term.
All the service provider needs to do is provide the option to rent out VM's with access to the inference cards.
Have the customer provide the model.
Maybe provide a bit of a speedup, by letting the customer mount the model by hash.
Sure, that all sounded way too technical 10 years ago, and somebody would have said that's already too much annoyance to deal with for average customers.
Reluctantly i'd agree.
That was 10 years ago. Nowadays, with some proper docs, this entire set up can be up and running in 5 minutes.
You could easily run it with SSD streaming on as little as 192GB RAM (The dense portion is around 115GB or so at native precision, and the remainder would fit KV and a cache for the most commonly referenced experts.) This fits Gorgon Halo, which is a reasonably priced platform. It would be slow, but still usable for high-value tasks.
Excuse me, could you elaborate how you came up with 192GB of RAM? I almost thought they downsized model, but it's still 2.8T tokens. Are we talking about not-yet-out 1-bit quant or assumption that something like ds4 (daskstar, not deepseek) will heavily optimize it?
All these very large models are sparse mixtures of experts (SMoE), where typically only 2-3% of weights are active for a given forward pass.
Usually there are some static experts in each attention layer, plus a whole bunch of dynamically-routed experts. Typically some form of LRU cache is used for the dynamic experts.
So you can keep only the static expert(s) and routing layers in memory and then stream the dynamic experts from SSD, at a modest inference speed cost driven by the latency required to load layers from SSD when they aren’t already in the memory cache.
The LRU cache works reasonably well because a given task will often hit the same experts on each iteration of the forward pass.
For example, on a Mac Studio, Qwen3.8-flash-next (180b total parameters) with 50% of experts in memory and n-gram offloading has about 1/4 to 1/3 the inference speed of everything in memory. That requires 32GB of vRAM for the weights (plus a little extra for context, obviously)
It's not based on loading the full 1.5TB model into VRAM or system RAM, it's just streaming parameters on demand from SSD - which is worthwhile since the MoE portion of the model is quite sparsely accessed. This has been demoed on a 128GB M5 MacBook Pro with lousy performance (0.3 tok/s or so) because it had to load at least ~25GB from SSD per token (comprising the entirety of the sparsely routed active params) and potentially even reload some of the dense weights, depending on what exactly managed to fit on that platform. With 192GB you'd avoid any reloading for the dense weights and even cache a few layer-experts.
Yes, I understand the principle, but will it be usable? E.g. in my personal opinion border of usefulness starts somewhere at 7-10 tok/s. I currently have 192GB and 24GB VRAM, and I can run Qwen-Flash-Next (4 bit) and GLM5.3-Flash (4 bit) at 25 tok/s and 7 tok/s respectfully (with quantized cache on beellama). Parallel queries worsen result significantly but can push aggregate to around 50 tok/s.
I can't, however, run DS4.1-1017, for example, but maybe I didn't find proper options and quantization yet.
So I'm really interested how to be sure what can/cannot fit in specific hardware (for example, I want to buy a Mac Studio with 64 or 128 GB RAM, what MoE model it would run at comfortable speed? Will it be qwen-flash-next or something bigger? And so on).
So far websites with "benchmarks" were really terrible in that regard (and huggingface calculator even worse!)
Wow, TIL. Is there any calculator or rule of thumb you use to figure out the expected performance of a model on some DRAM/SSD split? For instance I am wondering what the performance could be like on a 512GB M5 Mac Studio.
The rule of thumb is to just test it on your chosen platform, with your workload. SSD performance matters a lot obviously, you'll want to run PCIe 5.0 if at all feasible, or a recent M5 Pro/Max/Ultra or M6 Mac with high storage capacity (lower tiers have much worse performance). You can figure out a worst case easily (assuming that the fraction of sparse params you manage to cache in VRAM or system RAM has a fair chance of being picked among the total) but in practice you do quite a bit better than that - which is why even very small caches can be potentially worthwhile.
Due to costs I am unfortunately years away from ever seeing hardware at this level, but if you just want to study LLM's and AI then there are still plenty of models that work fine on crappy hardware, and you don't have to worry about them becoming sentient on you :-)
> a flood of 8x B300 will be homelab ready in a few months
Haha, that's beyond wishful thinking, I'm afraid. We're not even seeing a flood of A100s (6 yo and several generations old). Every GPU is still being squeezed right now, and the trend will probably continue for the foreseeable future.
Last I checked AWS still sells EC2 instances using 2018 era GPUs. Most people underestimate just how much long tail demand there is and how long these cards last, despite the higher power consumption.
Yes, but if data center space gets tight, it creates an incentive to replace old inefficient models with better ones, i.e. sqeueeze more performance out of limited data center space.
I agree, but it would have to be a place with plenty of energy and the only such place outside the US is a war zone.
Perhaps one option is building data centers along with solar power plants in sunny countries that have a lot of space. I'm not sure about the water/cooling situation though.
The rise of China due to hand-over-fist ceeding of power and influence fueled in equal parts by the right and left is so astonishing it'll certainly go down in the history books (written in Mandarin, of course).
Bonus points when the morons are doing so because they get all their political news from a Chinese platform. The [Americans] have gone insane.
A lot of the data center build out has been bipartisan. Unless Dems realize they need to listen to their constituents to avoid a 2028 bloodbath (they probably won't), I wouldn't count on that to save us from RAM/GPU prices.
No, also because that would mean I have to read up on rope scaling or whatever to go beyond the default 256k context, which so far I’ve avoided. But you can allocate it as 220k context for 2 parallel Pi workers, that really speeds things up! Or as ~8x 50k context for dedicated parallel document processing.
> Advancing the Frontier of Open Intelligence.
Too bad you probably need something like an 8x B300 node to run this. Black Friday is coming soon though, but I fear this open intelligence will be pacing anyone's liquidity frontier so I'll probably keep trailing the frontier for a while with nvidia/qwen3.8-27B-NVFP4 on dual RTX PRO 4000 in TP=2. I can almost fit 450k context in there.