This looks very interesting. Their new language modelling objective shows some large improvements over masked diffusion modeling.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.
https://arxiv.org/abs/2602.06036 - they use a faster (single-pass) diffusion model to generate predictions for 7 to 15 tokens in a single step. Then if they're accepted that really speeds up the autoregressive model because it doesn't have to generate tokens one at a time.
Just for additional context: You have to do more computational work to check them, beyond the extra work in the drafter, but you have much less of a memory bottleneck, which is extremely worth it on current hardware.
However, they hardly compare against the conventional next-token-prediction objective, which suggests they don't beat it, except for highly non-linear tasks like sudoku. Unfortunately, machine learning papers often tend to focus exclusively on the positives without mentioning bad aspects.