Autoregressive models usually produce one token at a time. That makes response speed depend on a series of target-model passes. Speculative decoding changes the rhythm, not the answer: a cheap model races ahead, then the expensive model checks a group of proposed tokens in parallel.
Speculative decoding is only faster when the target model accepts enough draft tokens to amortize the verification pass.
How does speculative decoding preserve output quality?
The draft model proposes a short run of tokens. The target model scores that whole run in one pass, accepts consecutive proposals that fit its own distribution, and resumes from the first rejection. The target remains the final authority on every emitted token. In the common exact version of the method, acceptance and correction rules preserve the same distribution as ordinary inference, rather than accepting a plausible shortcut.
The apparent paradox is hardware. A large model can often verify several positions together more efficiently than it can wait for one sequential decode at a time. A good draft therefore converts otherwise idle parallel work into accepted output tokens.
Serving software still needs to keep the draft and target states aligned after every accepted or corrected segment. That implementation work can erase a theoretical gain when request batches are small or the draft model is poorly matched.
What determines whether it actually speeds up a service?
Draft acceptance rate is the central metric. A weak draft model forces frequent corrections and adds overhead. A strong draft can make one target pass yield several final tokens. Measure accepted tokens per verification pass on representative traffic, not a hand-picked prompt. A draft model earns its place only when accepted tokens outweigh its verification overhead. Model size, sampling settings, batch size, and request shape all affect the result.
Is this the same as model routing?
No. Model routing chooses which full model should handle a request, often by cost or difficulty. Speculative decoding uses two models together inside one response. It also differs from a KV cache: cache reuse avoids recomputing earlier attention state, while speculation reduces the number of sequential decode cycles. For a fresh uncached request, neither reduces prefill or time to first token. Their effect is on decode-stage latency and streaming throughput after generation begins. Time to first token still depends on connection setup, queueing, and reading the prompt.
This separation matters when setting a performance target. Use speculative decoding when sustained output speed is the measured bottleneck. Reduce prompt size, queueing, or repeated-prefix work when the complaint is the initial wait.