Benchmark metrics
Queueing latency
Also known as queue latency, queueing time, queuing latency, scheduler wait time
In plain English
Queueing latency is time a request spends waiting for resources before the serving system can schedule its work.
Technical definition
Queueing latency measures the elapsed time a request waits at a defined admission or scheduling queue before it is selected for execution.
Typical unit
milliseconds or seconds of queue wait
Engineering details
Frontend waiting, engine admission, and later scheduler stalls can have different timing boundaries. State which interval a metric records. Client-observed time to first token also includes network delivery, prompt processing, and first-token generation, so it cannot by itself isolate time spent in the queue.
Why it matters
Near saturation, waiting can dominate the response delay even if kernels run at their usual speed. Faster prefill does not necessarily fix an overloaded queue; admission policy, request mix, and available capacity also determine waiting time.
How to read it in InferenceX
Use queue depth and server timing metrics, when present, to investigate TTFT changes across concurrency points. Do not subtract an unrelated prefill estimate from client TTFT and report the remainder as measured queueing latency.
See the concept in real benchmarks
All articlesAgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM