What does AgentX measure?

Traditional load tests often send unrelated single-turn prompts at a fixed request rate. AgentX instead keeps a configurable number of complete session trees alive. A root coding conversation may spawn child conversations, wait for them, and resume after they finish. Each turn carries the accumulated message history, creating the long shared prefixes and bursty fan-out seen in real agent systems.

The prompt text is synthesized, but the workload shape comes from recorded sessions: input and output lengths, prefix-sharing relationships, subagent topology, and inter-turn timing. This preserves the parts that matter to schedulers, routers, and prefix caches without replaying private user content.

Where does the workload come from?

The scenario uses SemiAnalysis public Weka-format agentic-coding trace corpora. For reproducible comparisons, choose a date-pinned corpus rather than the rolling alias. A _256k variant removes individual requests above the 256K-token range and is the better match for servers with a context window around that size.

The short semianalysis_cc_traces_weka name is the legacy corpus without subagents. Use a dated corpus or a with_subagents alias when you want AgentX fan-out behavior.

What does the scenario lock?

Passing --scenario inferencex-agentx-mvp selects agentic replay and enforces the settings that define a comparable run:

• Streaming is enabled so time to first token (TTFT) and inter-token latency (ITL) can be measured.
• ignore_eos:true makes the server generate the requested output length instead of stopping early.
• Recorded per-turn delays remain intact; only globally idle time is capped at 10 seconds.
• first_turn_prefix cache busting gives every replayed conversation a unique first-turn marker, preventing artificial cross-session cache hits while retaining reuse inside a session.
• A valid run lasts at least 900 seconds; the default is 1,800 seconds.
• Warmup primes a deep prefix before profiling begins.
• A random seed is always present and recorded in the artifacts.

What is the minimal command?

Replace the URL, model, corpus, and concurrency with values for your deployment. AIPerf fills in the scenario defaults and writes results under ./artifacts/. Pin --random-seed for exact replay sampling and reconstructed-dataset cache reuse. Add --tokenizer when the served model name is not a resolvable Hugging Face tokenizer.

aiperf profile \
  --scenario inferencex-agentx-mvp \
  --url http://localhost:8000 \
  --model YOUR_MODEL \
  --endpoint-type chat \
  --public-dataset semianalysis_cc_traces_weka_062126 \
  --concurrency 256

Why does AgentX use streaming chat completions?

The recorded workload is represented as multi-turn message arrays, which map naturally to an OpenAI-compatible chat-completions endpoint. Streaming exposes first-token and token-to-token timing; a non-streaming response cannot provide those core latency measurements.

--use-server-token-count changes only metric accounting. It trusts the server's usage fields when local tokenization differs because of a tokenizer revision or invisible chat-template overhead. It does not change prompt construction.

What does concurrency mean?

Concurrency is the number of live session trees, not the number of in-flight HTTP requests. A session can fan out into several child conversations, so instantaneous request concurrency may exceed the configured value.

There is intentionally no request-rate control in this scenario. Offered load is produced by the number of active sessions together with their recorded think times and fan-out patterns. This makes --concurrency the primary load dial.

Why can the first run take so long?

Before traffic starts, AIPerf downloads the corpus and reconstructs it into tokenized, cache-aware conversation trees. The result is stored in a memory-mapped disk cache. With the same corpus, tokenizer, reconstruction settings, entry count, and random seed, later runs can restore that prepared dataset in seconds.

For a cold run, raise both configuration timeouts together. The visible run then proceeds through dataset configuration, warmup, timed profiling, and drain/export. Benchmark duration covers only profiling, so total wall time is longer.

export AIPERF_DATASET_CONFIGURATION_TIMEOUT=1800
export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT=1800

Can I run a short smoke test?

A valid run cannot be shorter than 900 seconds. For a cheap but valid check, use low concurrency and keep the 900-second minimum. --num-dataset-entries reduces reconstruction work, but changes the workload and should not be used for a result you plan to compare or submit.

For a minutes-long connectivity test, use --unsafe-override with a short duration and accept the invalid stamp. Keeping the full corpus and pinning the random seed lets the subsequent valid run reuse the reconstructed dataset cache.

How do I know the load generator is not the bottleneck?

AIPerf distributes work across worker processes and reports worker health every two seconds. Treat the high-CPU warning as a saturation signal: add client cores or another load-generator host before trusting the result.

For deeper diagnosis, --show-trace-timing separates time waiting for a free pooled client connection from time waiting on the server's first byte. Healthy worker CPU and little connection-pool blocking are strong evidence that the measured limit is server-side.

Why do connection resets appear partway through a run?

Client and server keep-alive settings may disagree. If AIPerf retains an idle pooled connection longer than the server keeps it open, the client may reuse a closed socket. Set the client keep-alive below the server value. This often appears during paced profiling rather than warmup because warmup connections do not remain idle as long.

export AIPERF_HTTP_KEEPALIVE_TIMEOUT=4

How should multi-replica routing work?

Conversation-aware routing is essential when a router fronts several replicas. If turns from one conversation land on different workers, prefix-cache reuse collapses and the benchmark measures routing scatter rather than the serving stack's cache capability.

AIPerf assigns a stable correlation ID to each conversation; subagents receive their own stable IDs. These controls are not scenario-locked because they change request placement rather than request content.

• Prefix-aware routing matches message history against cached prefixes. SGLang Model Gateway provides cache_aware; Dynamo provides KV-aware routing with --router-mode kv.
• Sticky routing maps the AIPerf correlation ID to a router session header. AIPerf supports SGLang's X-SMG-Routing-Key, Dynamo session headers, and a generic X-Session-ID integration.

Which metrics should I inspect?

AgentX reports TTFT, ITL, end-to-end latency, output throughput, and error rates alongside prefix-cache behavior, context overflow, session and subagent execution, worker saturation, and scenario compliance.

Always check submission_valid before comparing runs. Excessive context overflow, cancellation, unsafe overrides, or other scenario violations can invalidate an otherwise complete artifact.

Practical rules of thumb

• Match the corpus to the server's context window; prefer the _256k corpus for a roughly 256K server.
• Use date-pinned corpora and a pinned random seed for comparisons.
• Keep cache busting enabled. Disabling it inflates cache-hit results when traces recycle.
• Preserve recorded timing. Compressing individual trace delays changes cache-TTL behavior and session overlap.
• Monitor client CPU and connection-pool wait time before calling a result server-bound.
• Use conversation-aware routing for every multi-replica deployment.
• Do not interpret synthesized prompt prose; the benchmark preserves token and cache structure, not semantic content.