Benchmark metricsRPS
Request rate
Also known as arrival rate, offered load, requests per second
In plain English
Request rate describes how quickly new requests arrive, rather than how many requests are already running or have finished.
Technical definition
Request arrival rate is the number of new requests offered or dispatched per unit of time, usually expressed in requests per second.
Typical unit
requests per second (requests/s)
Engineering details
A configured target rate, the actual client dispatch rate, and the rate observed by the server can differ. Network delays, client limits, and backpressure affect delivery. The same average rate can describe smooth arrivals or sharp bursts, and different prompt lengths can impose very different work at that rate.
Why it matters
Arrival rate sets demand; request throughput measures completions. If demand persistently exceeds capacity, requests accumulate, fail, or time out. Comparing only completed throughput can hide this overload and the associated latency.
How to read it in InferenceX
Read the workload’s arrival model alongside its sequence lengths and concurrency. AgentX client counts describe active sessions whose requests depend on prior completions and tool pauses; they do not specify a fixed requests-per-second load.
See the concept in real benchmarks
All articlesAgentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200