Benchmark metrics
Open-loop benchmark
Also known as open-loop load test, arrival-driven load
In plain English
An open-loop benchmark schedules new requests independently of when earlier requests finish, exposing what happens when arrivals exceed serving capacity.
Technical definition
An open-loop benchmark schedules request arrivals from an external rate or timestamp sequence rather than waiting for each client’s previous response.
Engineering details
Arrivals may be evenly spaced, sampled from a distribution, or replayed from timestamps. When service slows, the offered load can continue and the queue can grow. A client-side concurrency cap or backpressure can prevent scheduled arrivals from being dispatched, so the requested rate and actual send rate must be distinguished.
Why it matters
This load model exposes overload and queueing that a completion-driven client can hide by sending fewer requests as latency rises. Results depend on arrival burstiness and the duration of the measurement window, as well as the mean rate.
How to read it in InferenceX
AgentX uses dependent, closed-loop agent sessions. Do not interpret its client concurrency as an open-loop request rate or compare the two load models without identifying their arrival rules.
See the concept in real benchmarks
All articlesAgentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200