Latency
Also known as response latency, inference latency
In plain English
Latency is how long you wait. For a streamed answer, that includes both the wait before it starts and the pauses between later words.
Technical definition
Latency is elapsed time experienced by a request. In streaming LLM serving it must be decomposed because waiting for the first token and waiting between later tokens are different behaviors.
Engineering details
Time to first token captures queueing and prefill delay. Time per output token captures decode cadence after streaming starts. End-to-end latency also depends on output length, so a single aggregate latency number can hide the part users actually notice.
Why it matters
Low latency can require smaller batches or more parallel resources, which may reduce hardware utilization and increase cost. Good serving design chooses a latency service level and then maximizes throughput within it.
How to read it in InferenceX
InferenceX exposes workload shape and concurrency alongside interactivity. This keeps a high-throughput batch point from being mistaken for a low-latency serving point.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM