Decode
Also known as autoregressive generation, token generation
In plain English
Decode is the model writing its answer one token at a time after it has read the prompt.
Technical definition
Decode is the inference phase that generates output tokens autoregressively, normally one accepted token per sequence per model step.
Engineering details
Each new token depends on preceding tokens, which limits parallelism across time. The model repeatedly reads weights and the sequence’s KV cache, making decode especially sensitive to memory bandwidth, batching, and communication.
Why it matters
Decode controls streaming interactivity and often dominates the cost of long outputs. Techniques such as speculative decoding, MTP, quantization, and wide expert parallelism aim to reduce the work or time required per accepted token.
How to read it in InferenceX
InferenceX decode performance appears as tok/s/user and aggregate tok/s/GPU across concurrency. Output sequence length, batch shape, precision, and parallelism must match for a fair comparison.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool