Benchmark metricsICL
Inter-chunk latency
Also known as streaming chunk gap, chunk arrival latency
In plain English
Inter-chunk latency measures pauses between streamed messages arriving at the client, and each message may contain more than one token.
Technical definition
Inter-chunk latency is the elapsed time between consecutive qualifying response chunks received by a streaming client.
Relationship
chunk gap = current chunk arrival time - previous chunk arrival time
Engineering details
A server can buffer multiple generated tokens into one chunk, especially with speculative decoding or a configured streaming interval. Chunk arrival gaps then differ from token-normalized decode latency. An average TPOT can look good while users still see irregular bursts separated by long pauses.
Why it matters
Chunk timing describes delivery smoothness at the client boundary. It includes effects of buffering and transport as well as server scheduling. A percentile of chunk gaps and a percentile of per-request average token latency summarize different populations and should not share a label.
How to read it in InferenceX
Keep chunk-based timing separate from the selected InferenceX interactivity metric. Check the benchmark client’s content-chunk rules and token-count convention before comparing streamed results from different engines or speculative settings.
See the concept in real benchmarks
All articlesAgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM