Draft model
Also known as draft head, speculator
In plain English
A draft model is the small fast model in speculative decoding that guesses several upcoming tokens for the large model to verify in one pass.
Technical definition
A draft model is the lightweight proposal component of speculative decoding, producing candidate token sequences that the target model verifies in a single batched forward pass.
Engineering details
Drafts take several forms: a separate small model from the same family, extra prediction heads trained onto the target model as in EAGLE style methods, or the multi token prediction heads some models ship with. The draft races ahead a few tokens cheaply, the target checks all of them at once, and accepted tokens are emitted together. Rejection falls back to the target model output, so results match the target distribution.
Why it matters
Draft quality sets acceptance length, and acceptance length sets the speedup. A well matched draft can multiply decode speed at low batch sizes, while a mismatched or overly aggressive draft wastes verify compute and can even slow serving under load.
How to read it in InferenceX
InferenceX records the speculative method and acceptance length behind each result and publishes golden acceptance length distributions for reproduction, because an unrealistic acceptance rate is a classic way a benchmark number stops describing production behavior.
Source material
See the concept in real benchmarks
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.