On: Quantized Llama models with increased speed and a ...

[qeternity]

> with tolerably close accuracy.

No, speculative decoding has exactly the same accuracy as the target model. It is mathematically identical to greedy decoding.

[kgc → qeternity]

Is there a reference for this? I was wondering the same thing.

[qeternity → kgc]

Read the original whitepaper or go look at how any framework implements it.

You will see that tokens not predicted by greedy sampling of the target model are rejected. Ergo, they are mathematically identical.