2026 has been the year where spec-dec has matured, it has been adopted by all big OSS engines and i'm sure it's present in quite a lot of inference providers as the default
I feel like P/D dissaggregation will be the next big one for providers, as prefill tends to be compute bound while decode mem bound which I guess each will have a different type of node
I feel like P/D dissaggregation will be the next big one for providers, as prefill tends to be compute bound while decode mem bound which I guess each will have a different type of node