Speculative decoding
Also known as spec decode, draft-and-verify decoding
In plain English
Speculative decoding lets a cheaper helper draft several tokens ahead, then asks the full model to approve them together instead of generating each one separately.
Technical definition
Speculative decoding proposes several future tokens cheaply and verifies them together with the target model, reducing the number of expensive serial decode steps.
Engineering details
A draft model or built-in prediction heads generate candidates. The target model evaluates those candidates in a batched verification pass and accepts the valid prefix without changing the target distribution when the algorithm is implemented exactly.
Why it matters
The speedup depends on how many draft tokens are accepted and on the cost of drafting and verification. Dense and MoE models can behave differently because verifying several positions may activate more expert weights.
How to read it in InferenceX
Compare speculative recipes at realistic acceptance rates and verify model quality. InferenceX distinguishes MTP-enabled and disabled curves because the benefit changes across concurrency and interactivity.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time — Huawei, GB300 NVL72, MI355X, B200
Day 0 Inference Performance, InferenceX, 100x performance improvement in 26 Days, Cost per Million Tokens, Huawei 950DT Inference Trace Analysis
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K