Closed-loop benchmark
Also known as closed-loop load test, closed-loop workload
In plain English
In a closed-loop benchmark, each simulated user waits for one step to finish before sending the next step in that session.
Technical definition
A closed-loop benchmark generates new work from each client in response to completion of its previous dependent request, subject to the workload’s recorded delays and branch structure.
Engineering details
Concurrency is the number of active clients or sessions; the simultaneous request count changes over time. Faster systems complete turns sooner and therefore issue more requests during the same profiling period. The exact request mix can vary slightly because progress through each sampled session depends on completion time.
Why it matters
This load model resembles interactive agents, where the next action depends on the previous result. Throughput and latency remain coupled: a faster response advances the session and creates later work sooner. Low-concurrency runs can show more sampling variation than large pooled runs.
How to read it in InferenceX
AgentX uses closed-loop concurrency. Its concurrency value is the number of agent clients; request batch size changes as the sessions advance. Read throughput, time to first token, and interactivity together.
Source material
See the concept in real benchmarks
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM