Chunked prefill
Also known as split prefill, piecewise prefill
In plain English
Chunked prefill reads a long prompt in slices instead of all at once, so other users keep receiving tokens while it is being read.
Technical definition
Chunked prefill splits prompt processing into fixed-size token chunks that the scheduler interleaves with ongoing decode work.
Engineering details
An unsplit prefill occupies the accelerator for as long as the whole prompt takes, and every user already streaming waits behind it. Splitting the prompt lets the scheduler alternate, so decode continues between chunks. The chunk size is a tuning knob: larger chunks prefill more efficiently, smaller chunks interrupt decode less.
Why it matters
The technique converts a first-token problem for one user into a small, steady tax on everyone else, which is usually the better trade. It matters far more as prompts grow, since a single unsplit hundred-thousand-token prefill would stall a deployment outright.
How to read it in InferenceX
Chunk size is part of the recipe and can differ across points on the same curve, so a jump in throughput may reflect a retune rather than new hardware. Long agentic prompts make the setting consequential in a way fixed short-prompt scenarios never expose.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool