Multi-token prediction
Also known as multi-token prediction heads
In plain English
MTP lets the model guess several upcoming tokens at once and then verify them, reducing the number of slow one-token-at-a-time steps.
Technical definition
Multi-token prediction uses auxiliary heads trained with the model to propose multiple future tokens for speculative verification.
Engineering details
Unlike a separate draft model, MTP proposals come from the target model’s own representation. This can improve proposal alignment and simplify deployment, but it requires a checkpoint trained with compatible MTP modules and engine support for the verification path.
Why it matters
MTP can exchange otherwise underused compute for fewer memory-bound decode steps. Gains are largest when draft acceptance is high and verification fits into available compute; at large batches the extra work may provide less benefit.
How to read it in InferenceX
InferenceX reports MTP as a recipe dimension. Acceptance rate or acceptance length, workload distribution, numerical quality checks, and matched interactivity all matter when translating a benchmark gain to production.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time — Huawei, GB300 NVL72, MI355X, B200
Day 0 Inference Performance, InferenceX, 100x performance improvement in 26 Days, Cost per Million Tokens, Huawei 950DT Inference Trace Analysis
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K
AMD MI355X GLM-5 Inference: Up to 40% Cheaper per Million Tokens than B200 on SGLang FP8
14 weeks after GLM-5 launched, AMD landed both MTP and non-MTP SGLang FP8 recipes on MI355X — fused MLA + FP8 KV cache via TileLang flips the single-node FP8 cost curve in AMD favor across most of the performance Pareto