On DeepSeek V4 Pro, NVIDIA's most competitive solutions are GB300 Dynamo TRTLLM and GB200 Dynamo vLLM. Both configurations rely on prefill-decode disaggregation to achieve high throughput at reasonable interactivity. GB300 configurations additionally employ wide expert parallelism decode instances at DEP32 in order to achieve higher throughput in the middle of the frontier.

The two points are closer on token rate than on first token

The 2xDEP8+1xDEP12 GB200 point is significantly closer to the 3xDEP8+1xDEP16 GB300 point in terms of tokens per second per user than it is in terms of time to first token. Reading only the interactivity axis would therefore understate how differently these two configurations behave.

TTFT is, in general, more sensitive to the spikiness of the workload. The GB300 point achieves much higher overall concurrency, so it incurs more subagent traffic and hence more cold prefills. A subagent starts with fresh context and no reusable prefix, which means its first request is a full prefill rather than a cache hit, and enough of them arriving together is what moves the tail.

Why this is an agentic effect rather than a hardware one

None of this appears on a fixed-sequence workload. At 8k1k every request is independent and identically shaped, so concurrency scales the batch without changing the mix of cached and uncached work. On AgentX, raising concurrency changes the composition of the traffic itself: more sessions means more parallel subagent branches, and those branches arrive in bursts rather than as a smooth stream.

The practical reading is that a rack-scale configuration tuned to sit further right on the throughput curve is also accepting a burstier prefill load, and the cost of that decision lands on first-token latency rather than on the token rate the same chart advertises.

These results are one slice of AgentX 1.0. The full analysis, the replay methodology, and the 70+ upstream PRs the benchmark drove are in AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?. Every point is explorable on the free dashboard.