Serving
Decode admission control
Also known as decode concurrency cap, decode request limit
In plain English
Decode admission control limits how many requests enter a decode worker, trading faster generation for admitted requests against waiting time for those still queued.
Technical definition
Decode admission control is the policy that determines when a request may occupy a decode worker’s active capacity, often using a request-count or resource limit.
Engineering details
A request may wait at the router or engine until an eligible worker has room. A lower cap can reduce the active batch and improve per-user generation speed, while a higher cap can increase throughput through batching. The location of the queue determines which latency interval includes the wait. A router cap is distinct from client concurrency and from a physical KV-memory limit.
Why it matters
A fast decode curve can coexist with slow time to first token when requests wait outside the worker. Evaluate admission policies with queueing, TTFT, end-to-end latency, and completed work under the same offered load.
How to read it in InferenceX
The TPU article’s llm-d Qwen3.5 comparison varies the number of requests admitted to a decode host. It explicitly discusses the resulting TTFT and end-to-end trade-offs, so its fastest decode point should not be treated as the fastest complete response.