Benchmark metrics
Request queue depth
Also known as waiting requests, pending requests, scheduler queue
In plain English
Queue depth counts requests waiting for service, which can rise even while the accelerators remain busy producing tokens.
Technical definition
Request queue depth is the number of requests waiting at a specified scheduling or admission boundary at a given time.
Typical unit
waiting requests at the sampled boundary
Engineering details
A deployment can have separate queues at the frontend, router, and engine. A server’s waiting-request metric describes its own boundary, not every outstanding client request. Requests may wait for compute, available KV-cache blocks, or a compatible batch. A sampled queue depth is an instantaneous count rather than a duration.
Why it matters
A growing queue can explain rising time to first token without a change in prefill kernel speed. Brief spikes may reflect subagent bursts; a sustained increase can indicate that arrivals exceed the system’s ability to admit work.
How to read it in InferenceX
AgentX detailed-point telemetry includes request queue depth when available. Read it alongside running requests, KV-cache utilization, and throughput, and check which engine exported the metric before comparing values across serving stacks.
See the concept in real benchmarks
All articlesAgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure