Tail latency
Also known as p90, p99, percentile latency
In plain English
Tail latency describes the slowest requests rather than the typical one, because the unlucky few are what users actually notice.
Technical definition
Tail latency is the latency at a high percentile of the request distribution, such as p90 or p99, rather than the mean or median.
Engineering details
A percentile answers a different question from an average. A p90 of five seconds means one request in ten waited at least that long, and that request may be the one blocking an agent from continuing. Distributions in real serving are heavily skewed, so the mean can sit far below the tail and hide it completely.
Why it matters
Capacity planning is usually written against a percentile, not an average, because a service that is fast on average and slow at the tail still fails its users. Optimizations can also improve the mean while worsening the tail, which a single-number summary would report as an unambiguous win.
How to read it in InferenceX
InferenceX reports percentile-qualified metrics and labels the percentile on the axis, so p90 TTFT and mean TTFT are never mixed on one comparison. Agentic runs are especially skewed, because end to end latency scales with output length and the longest generations dominate the tail.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost