Continuous batching
Also known as in-flight batching, dynamic batching, iteration-level scheduling
In plain English
Continuous batching lets new requests join a running batch the moment old ones finish, instead of waiting for the whole batch to complete.
Technical definition
Continuous batching is a scheduling technique that admits and retires requests at every generation step, keeping the batch full as individual sequences finish at different times.
Engineering details
Static batching waits to collect a group of requests, runs them together, and returns them together, so a batch runs as long as its slowest member. Because LLM outputs vary wildly in length, that wastes enormous capacity. Continuous batching reforms the batch every iteration: a sequence that emits its final token leaves immediately and a queued request takes its slot at the next step, so the accelerator stays saturated.
Why it matters
This is one of the foundational optimizations of modern LLM serving and a large part of why open source engines displaced naive deployment. It multiplies throughput at a given latency and pairs naturally with paged KV cache memory, which makes slot reuse cheap.
How to read it in InferenceX
Every engine InferenceX benchmarks, including vLLM, SGLang, and TensorRT-LLM, uses continuous batching. Concurrency sweeps measure how well each scheduler holds interactivity as the batch fills, which is where implementation differences between engines become visible.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool