Benchmark metrics
Request throughput
Also known as completed requests per second, completion rate
In plain English
Request throughput counts finished requests per second; token throughput counts how much generated text the system delivers in that time.
Technical definition
Request throughput is the number of successfully completed requests divided by the elapsed time in a specified measurement window.
Relationship
request throughput = successful completed requests / measurement seconds
Engineering details
A short response and a long response each count as one completed request. Completion rate therefore changes with the request mix even when token throughput stays similar. Report how failures, timeouts, warmup, and requests crossing the measurement boundary are handled; different window rules change the result.
Why it matters
A high completion rate can come from serving shorter requests rather than a faster system. It also does not establish that completed requests met their latency targets. Goodput adds an explicit service-level requirement to the counted work.
How to read it in InferenceX
InferenceX throughput curves primarily compare token rates, with the selected normalization shown on the axis. Do not relabel tokens per second as requests per second or interpret completed AgentX requests as completed agent sessions.
See the concept in real benchmarks
All articlesInferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200