Agentic Benchmark for LLM Inference: Metrics and Methodology

An agentic benchmark for LLM inference measures how a model-serving system behaves under the traffic generated by AI agents. It does not grade whether an agent fixed a bug or completed a workflow. It measures the infrastructure serving each model call: latency, throughput, cache reuse, concurrency, and cost.

That distinction changes the workload. A conventional inference benchmark can send independent prompts with fixed input and output lengths. An agent benchmark must represent sessions in which context grows across turns, requests share long prefixes, tools insert pauses, and subagents create parallel branches. The next request often cannot start until an earlier request finishes.

AgentX is InferenceX's open benchmark for this workload. It converts privacy-preserving shapes from opt-in coding-agent traces into deterministic AIPerf replays, then compares complete hardware and software serving configurations across concurrent agent clients.

What is an agentic inference benchmark?

An agentic inference benchmark is a systems benchmark for multi-turn agent traffic. The system under test includes the accelerator, server topology, inference engine, model checkpoint, precision, parallelism strategy, cache configuration, and request scheduler.

It answers questions such as:

• How many concurrent agent sessions can the deployment serve?
• How long does each agent wait for its first token?
• How quickly does each active agent receive output tokens?
• How much output throughput does each chip deliver at a target latency?
• Do repeated prefixes hit HBM, host memory, or recompute paths?
• What does a million served tokens cost at a usable operating point?

The word “benchmark” is also used for agent task evaluations, so the scope must be stated. The two test types answer different questions:

AgentX covers the second row. Semantic task quality requires a separate evaluation.

Why fixed-sequence inference benchmarks miss agent traffic

Fixed-sequence benchmarks remain useful for controlled comparisons. An 8K input and 1K output workload isolates a common request shape and makes configurations easy to reproduce. It does not reproduce a long-running agent session.

These differences can change which system wins. More HBM can keep long-lived KV cache resident. Better prefix caching can avoid reprocessing hundreds of thousands of prompt tokens. Scheduler and router costs that are small for independent prompts can grow with the number and length of live sessions.

An agent benchmark is a request graph

An agent session is not a flat prompt list. Main-agent turns form a dependency chain. A turn can spawn one or more subagents, and the main chain may wait for those branches before it continues. Auxiliary calls can run without joining the main path.

AgentX replays this graph in a closed loop. Each client sends its next eligible request only after the required earlier work completes. Faster servers advance through a session sooner and create later work sooner. The instantaneous HTTP request count can exceed client concurrency when subagents fan out, then fall when branches finish.

This is why AgentX concurrency means live agent clients, not a fixed request batch. A run at concurrency 50 can place more than 50 requests in flight if several clients have active subagents.

How AgentX builds a reproducible workload

AgentX turns recorded session structure into a content-free replay in four stages.

1. Capture request shape and timing

An opt-in HTTP proxy records request arrival and completion times, input and output token counts, conversation IDs, and subagent IDs. It does not publish prompts, source code, tool arguments, or tool results.

2. Preserve prefixes without preserving content

Each input becomes a chain of session-scoped hashes in 64-token blocks. Matching block IDs retain the prefix relationships between turns. AIPerf replaces those blocks with deterministic synthetic coding and tool-use tokens before replay.

3. Reconstruct dependencies

Main-agent requests become a linear chain. Subagent requests become separate chains with spawn and join dependencies. Recorded inter-turn gaps reproduce the time spent using tools or waiting for other work. The replay keeps client-visible ordering and overlap without claiming access to provider-internal events.

4. Replay the same scenario across systems

A fixed seed controls session sampling, replay start points, and synthetic payloads. Each hardware and software configuration receives the same scenario definition, so differences in the resulting curve come from the serving system rather than a hand-written prompt batch.

AgentX v1.0 contains 393 published sessions selected from opt-in Claude Code traces. Each selected session has at least 20 requests and no more than 10 concurrent subagents.

The full variant retains contexts up to 1M tokens. The 256K variant removes requests above its context cap while preserving the timing and subagent overlap of the requests that remain. The AgentX methodology publishes the dataset rules, figures, and source links.

Benchmark controls that make comparisons valid

Long-running agent workloads are sensitive to starting state, cache contents, speculative decoding, and host memory. AgentX locks those variables with explicit controls.

Cache warmup and measurement window

A fixed seed selects a starting point between 25% and 75% of each recorded session. A max_tokens=1 primer materializes active main-agent and subagent prefixes. Each replay lane then completes 10 additional warmup requests before measurement starts.

Only the following one-hour profiling window contributes to reported metrics. Each recycled replay receives a unique cache-bust marker so unrelated sessions cannot accumulate a shared prefix.

Synthetic payloads and speculative decoding

Synthetic tokens preserve length and prefix structure, but they do not naturally reproduce draft-token acceptance. AgentX therefore uses an acceptance length measured from the coding category of SPEED-Bench for each model, speculative method, draft length, and thinking mode. The value is recorded in versioned golden files and applied through engine controls.

This keeps the comparison focused on serving performance instead of accidental differences in how synthetic text interacts with a speculator.

DRAM and KV-cache offload

Host-memory capacity can change whether a long prefix is reloaded or recomputed. Servers without a standardized DRAM configuration are capped at 3 TB. Standard systems such as GB200 NVL72, GB300 NVL72, and TPUv7 can use their installed capacity. A benchmark configuration can access only the host-memory share proportional to its GPU allocation.

The replay format and dependency rules are documented in the AIPerf WEKA trace guide.

Metrics for an agentic benchmark

One number cannot describe agent serving. Report throughput, latency, and interactivity together across a concurrency sweep.

Percentiles matter. A low mean TTFT can coexist with a poor p90 or p99 experience if a subset of agent branches waits behind long prefills. Aggregate output throughput can rise while each individual client receives tokens more slowly.

Read the full throughput-interactivity curve

AgentX sweeps concurrent agent clients and plots the resulting operating points. Low concurrency usually favors per-client responsiveness. Higher concurrency can improve batching and aggregate throughput until queueing, memory pressure, or another bottleneck reduces interactivity.

The useful result is the Pareto frontier, not the single point with maximum throughput. Hardware A may produce the highest tok/s/chip at low interactivity while hardware B leads at the response speed required by an interactive coding product.

Closed-loop replay also means a faster configuration can progress farther through its sampled sessions during the same profiling hour. The exact request mix can vary slightly, especially at low concurrency. Results should include the full curve and enough concurrency to reduce sampling noise.

What agent traffic stresses in an inference stack

Agentic inference exposes several bottlenecks that fixed prompt batches can understate.

• KV-cache capacity: Long contexts and many live sessions keep large caches resident for longer.
• Prefix-cache correctness: A false hit produces wrong state; a missed hit repeats expensive prefill work.
• Offload bandwidth: Moving a long prefix from host memory can be faster than recomputing it, but transfer granularity and descriptor overhead matter.
• Scheduler fairness: Parallel subagents can raise throughput while an individual branch waits too long.
• Router and frontend CPU: Prefix matching, expiry tracking, serialization, and streamed response handling scale with live session state.
• Long-context kernels: Attention, sparse attention, normalization, and graph-capture choices can behave differently across varying sequence lengths.
• Speculative decoding: Acceptance controls and scheduler lifetime affect both throughput and interactivity.

The AgentX optimization tracker documents concrete changes in vLLM, SGLang, TensorRT-LLM, ATOM, Dynamo, and KV-cache systems that were exposed by this workload.

How to compare AgentX results

Use the same scenario contract on both sides of a comparison:

1. Match the model, checkpoint, precision, and thinking mode.
2. Match the AgentX dataset variant and context limit.
3. Record the inference engine, container image, parallelism, and speculative-decoding method.
4. Apply the same golden acceptance-length rule.
5. Match DRAM capacity, KV-cache offload policy, and the proportional memory rule.
6. Use the same seed, warmup procedure, profiling duration, and client-concurrency sweep.
7. Compare per-chip and whole-system results when system sizes differ.
8. Compare at the same p90 interactivity or latency target.
9. Include failed and unstable configurations instead of reporting only surviving points.

An iso-interactivity comparison is usually more useful than peak throughput. It answers a deployment question directly: at the response speed users need, which system serves more agent work per chip or per dollar?

What this agent benchmark can and cannot answer

AgentX can measure serving capacity, responsiveness, cache reuse, memory pressure, throughput, and cost under a trace-derived agent workload. It can show whether a software change improves long-context multi-turn serving on the same hardware, or whether a different hardware topology changes the useful frontier.

AgentX cannot tell whether the model wrote correct code, selected the right tool, followed a business policy, or completed the source task. The synthetic payload deliberately removes those semantics. Use a task evaluation for agent quality and AgentX for inference performance. Keep the scores separate because they describe different systems questions.

Frequently asked questions

What is an agentic benchmark for LLM inference?

It is a systems benchmark that measures how an inference stack serves multi-turn agent traffic. It replays growing contexts, shared prefixes, tool delays, and subagent branches, then reports latency, throughput, interactivity, cache behavior, capacity, and cost.

How is an agentic benchmark different from a fixed-sequence inference benchmark?

A fixed-sequence benchmark sends independent requests with a specified input and output length. An agentic benchmark replays dependent sessions whose request lengths, timing, prefix reuse, and parallel branches change over time. The latter measures the serving conditions created by working agents.

What does the AgentX benchmark measure?

AgentX measures complete inference configurations under trace-derived coding-agent traffic. It sweeps concurrent agent clients and reports metrics including output throughput per chip, p90 interactivity, TTFT, ITL, cache behavior, and serving cost.

Why does AgentX use closed-loop concurrency?

An agent cannot issue every turn in advance because later actions depend on earlier results. In closed-loop replay, each client sends its next eligible request after its dependencies complete. This preserves the relationship between server speed and how quickly a session creates more work.

Which metrics matter most for agent inference?

Use output throughput, p90 interactivity, TTFT, ITL, cache-source distribution, queue depth, and cost together. Compare full concurrency curves at the same latency or interactivity target. A maximum-throughput point alone does not describe an interactive deployment.

Does an agent benchmark measure model or agent quality?

Some benchmarks do, but AgentX does not. AgentX measures inference-system performance with synthetic payloads that preserve workload shape. Task correctness, tool selection, and policy compliance require a separate agent evaluation.

All articles and posts are © SemiAnalysis. All rights reserved. The AGPL-3.0 license covering the application source code does not apply to article content.