Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.

However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.

 help



Maybe if it's worse than next-token-prediction it could still be used as a drafting model, similar to how DFlash works.

What's that?

https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.

Just for additional context: You have to do more computational work to check them, beyond the extra work in the drafter, but you have much less of a memory bottleneck, which is extremely worth it on current hardware.

I'm pretty sure fast speculative decoding models need to have the same objective (NTP, MDM, NCM) as the larger model they try to accelerate.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: