Serving
Request preemption
Also known as preemption, request recomputation, scheduler preemption
In plain English
Preemption interrupts an in-progress request so other work can run, sometimes requiring its attention state to be rebuilt before it resumes.
Technical definition
Request preemption suspends a scheduled request and reclaims resources needed by other requests, with the suspended work resumed according to the engine’s policy.
Engineering details
KV-cache pressure can force a serving engine to stop a sequence that has already begun generating. The engine may recompute discarded state or restore saved state before continuing. This differs from evicting an unused prefix because an active request now loses progress time, even if its previously emitted tokens remain valid.
Why it matters
Repeated preemption adds work and can create long response stalls. High accelerator utilization during recomputation does not imply useful progress at the same rate. Counts of preemptions and request latency together help identify this failure mode.
How to read it in InferenceX
Inspect engine telemetry and logs when a concurrency sweep suddenly loses interactivity or throughput. Do not assume every stack exports the same preemption counter or infer preemption solely from a high KV-cache utilization reading.
See the concept in real benchmarks
All articlesAgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set
50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from