Prefill
Also known as prompt processing, context encoding
In plain English
Prefill is the model reading and understanding your prompt before it begins writing the answer.
Technical definition
Prefill is the first inference phase, in which the model processes the input prompt and populates the KV cache before generation begins.
Engineering details
Prompt tokens can be processed in parallel, producing large matrix operations that are usually compute intensive. Prefill cost grows with input length and contributes heavily to time to first token.
Why it matters
Prefill has a different resource profile from decode. When both share the same workers, large prompt jobs can interrupt decode batches and make streaming latency less predictable.
How to read it in InferenceX
Disaggregated recipes place prefill on a separately sized GPU pool. When reading a result, check the prefill tensor parallelism, GPU count, input length, and whether KV state must cross a network before decode.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4