TPU v7 (Ironwood) vs GB300 NVL72: AgentX Results

Summary

Method

TPU v7 has AgentX results on two models, Qwen3.5-397B and Kimi K3, and the dashboard lists it as TPU7x. The x-axis is p90 interactivity (1 / p90 inter-token latency) in tok/s/user. Matched-interactivity values are interpolated along each system's Pareto frontier with no extrapolation. Throughput counts input tokens, including KV-cache hits, and output tokens. A TPU v7 chip is counted as one chip even though it exposes two devices to the framework, so throughput is per physical chip on both sides.

Tokens per dollar, the dashboard's default metric, counts total tokens (input and output) served per $1 of TCO. The cost is the SemiAnalysis AI Cloud TCO Model owning cost at large hyperscaler volume: $2.31/GPU/hr for GB300 NVL72, and two figures for TPU v7. The external figure, $1.21/chip/hr, is the owning cost for a buyer other than Google, and renting from Google Cloud on a 3-year commit costs $2.00/chip/hr. The dashboard default is $1.03/chip/hr, Google's own cost to own and run the chip. The tables show both as ext ($1.21) / int ($1.03). Only TPU v7's cost differs between the two bases, so GB300 NVL72 and the other GPUs have the same tokens per dollar on both. The charts use the dashboard default, so on the external cost every TPU v7 point sits 15% lower than plotted.

Per-MW figures use the all-in provisioned power per chip from the SemiAnalysis Datacenter Industry Model, 1.207 kW for TPU v7 and 2.12 kW for GB300 NVL72, which are planning figures for deployed systems. A TPU v7 chip is provisioned for 57% of a GB300 GPU's power and costs 52% as much per hour on the external cost and 45% on the internal cost, so per-dollar ratios run 9% above per-MW ratios on the external cost and 28% above on the internal cost.

Our Vera Rubin NVL72 results describe the AgentX workload.

Charts load live dashboard data and can include newer runs than the tables and numbers in the text.

Specs

TPU v7 values are from Google Cloud's TPU7x documentation, which lists inter-chip interconnect bandwidth as 1,200 GB/s per chip across both directions. Blackwell Ultra values come from the InferenceX GPU specs.

A fully bandwidth-bound decode step would cap TPU v7's per-dollar advantage at the HBM bandwidth ratio over the hourly cost ratio, 0.92x / 0.52x ≈ 1.8x on the external cost and ~2x on the internal cost. The per-dollar ratios in the Qwen3.5 table exceed that at 200, 300 and 400 tok/s/user on both bases, because GB300 NVL72's FP8 curve falls away faster than TPU v7's in that band and has a gap in coverage there. Just above 200 tok/s/user, where TPU v7's frontier drops from its 22-agent to its 18-agent point, the ratio dips to ~1.6x on the external cost and ~1.9x on the internal cost.

Qwen3.5-397B FP8

TPU v7 delivers more tokens per $1 TCO than GB300 NVL72 FP8 from ~103 tok/s/user on the external cost and at every matched point on the internal cost, where its smallest lead is 0.3% at 81.5 tok/s/user, the low end of its curve. It trails per MW until ~117 tok/s/user and per chip until ~270 tok/s/user. GB300 NVL72's FP8 throughput falls 54% from 100 to 200 tok/s/user while TPU v7's falls 14%.

Qwen3.5-397B-A17B has 397B total and 17B active parameters, with three Gated DeltaNet linear-attention layers for every gated full-attention layer (model card; see the model page). TPU v7 runs vLLM with FP8 and MTP on one 4-chip host, aggregated, with KV-cache offload to host memory from 12 agents up. GB300 NVL72 runs Dynamo SGLang with FP8 and MTP on 4 to 32 GPUs, mostly disaggregated.

The chart below shows total tokens per $1 TCO for both systems on Qwen3.5-397B, with GB300 NVL72's FP4 lines alongside FP8:

As you can see, GB300 NVL72's FP8 frontier has no measured point between 192 tok/s/user (180 agents) and 450 tok/s/user (44 agents), so its values at 300 and 400 tok/s/user in the table below are interpolated across that gap. A straight line between those two points would put GB300 NVL72 at ~26,700 tok/s/GPU at 300 tok/s/user, which would make TPU v7's lead there 1.81x per dollar on the external cost instead of 2.02x, and 2.12x instead of 2.37x on the internal cost.

Values are M tokens per $1 TCO, and ratios are TPU v7 / GB300 NVL72 FP8 on Dynamo SGLang.

TPU v7's largest per-dollar ratio to GB300 NVL72 FP8 is at ~559 tok/s/user, the 4-agent point at the top of TPU v7's curve: 5.67x per dollar on the external cost, 6.66x on the internal cost and 5.22x per MW. Above 450 tok/s/user GB300 NVL72's FP8 frontier drops from 44 agents to single-agent points at 510 tok/s/user on 4 GPUs and 581 tok/s/user on 8 GPUs, and the lead grows from 2.24x to 5.67x on the external cost. At ~559 tok/s/user GB300 NVL72 FP4 on Dynamo TensorRT-LLM delivers 1.56x TPU v7's tokens per $1 TCO on the external cost and 1.33x on the internal cost, between frontier points with 45 agents at 402 tok/s/user and 136 agents at 571 tok/s/user.

At peak throughput GB300 NVL72 FP8 serves 101,367 tok/s/GPU at 68 tok/s/user with a 32-GPU disaggregated config at 1,000 agents, or 158.0M tokens per $ on both bases. TPU v7's peak is 40,363 tok/s/chip at 81 tok/s/user with 32 agents on one host, or 120.1M tokens per $ on the external cost and 141.1M on the internal cost. That is 0.76x GB300 NVL72's peak per dollar on the external cost, 0.89x on the internal cost, 0.40x per chip and 0.70x per MW. At 32 agents the TPU v7 server's prefix-cache hit rate was 88.8% against a theoretical 96.4%, with HBM KV-cache usage at 93%. TPU v7's p90 time to first token stays between 0.35 s and 0.59 s across its frontier.

Our TPU externalization article reported up to 50% better performance per dollar for TPU v7 than B200 and B300 on the 8k/1k workload, with aggregated FP8 serving on all three and external TPU TCO. On AgentX, a different workload, TPU v7 delivers 1.80x the tokens per dollar of B200 Dynamo SGLang, the best B200 or B300 FP8 line per dollar there, at 150 tok/s/user and 2.34x at 200 on the external cost, and 2.12x and 2.75x on Google's internal cost. Per chip it serves 1.24x B300 SGLang at 150 tok/s/user and 1.62x B300 Dynamo SGLang at 200, the strongest B200 or B300 FP8 lines there.

Against GB300 NVL72 FP4

Ironwood has no native FP4 compute. Our LinkedIn post on NVIDIA's first GB300 FP4 counter-submission said it was "1.64x worse in performance per dollar than TPUv7 for external customers across most of the end-to-end performance curve." The post's chart used end-to-end latency on its x-axis. On the p90 interactivity axis used here, GB300 NVL72 FP4 on Dynamo TensorRT-LLM delivers 1.34x to 2.15x the tokens per $1 TCO of TPU v7 FP8 from 150 to 400 tok/s/user on the external cost, 1.14x to 1.83x on the internal cost and 1.5x to 2.3x the tokens per MW. Per MW it leads at every matched point.

The highest-throughput point on GB300 NVL72's FP4 Dynamo TensorRT-LLM line sits above 100 tok/s/user, so the table starts at 150. Values are M tokens per $1 TCO, and ratios are GB300 NVL72 FP4 / TPU v7 FP8.

Against GB300 NVL72's Dynamo SGLang FP4 line, TPU v7 FP8 delivers more tokens per dollar above ~215 tok/s/user on the external cost and above ~160 on Google's internal cost, and more per MW above ~230.

End-to-end latency includes time to first token. Against p90 end-to-end latency, TPU v7's Qwen3.5 frontier spans 3.4 s to 11.1 s, and each GB300 NVL72 FP4 Dynamo TensorRT-LLM frontier point in that span delivers fewer tokens per $1 TCO on the external cost than a TPU v7 point with lower latency. The TensorRT-LLM line's two highest-throughput points, at 565 and 704 agents, have p90 time to first token of 5.6 s and 8.1 s and p90 end-to-end latency of 11.9 s and 21.6 s. On the p90 interactivity axis they sit at 253 and 119 tok/s/user, inside TPU v7's range.

Kimi K3

TPU v7's largest per-dollar ratio on Kimi K3 is 1.15x on Google's internal cost at ~100 tok/s/user, where GB300 NVL72's frontier runs between 48-agent disaggregated configs on 24 and 32 GPUs and TPU v7's runs between its 32- and 16-agent points. TPU v7 also leads on the internal cost by up to 4% from ~230 to ~290 tok/s/user, where both curves are sparse, and GB300 NVL72 leads elsewhere on the internal cost by up to 1.34x. Across the matched range, GB300 NVL72 delivers 1.03x to 1.58x TPU v7's tokens per dollar on the external cost and 1.1x to 1.7x its tokens per MW.

Kimi K3 has 2.8T total and 104B active parameters, with Kimi Delta Attention in 69 of its 93 layers and Gated MLA in the rest (model card; see the model page). It ships with MXFP4 weights and MXFP8 activations, which matches the FP4 label on both systems' runs. TPU v7 runs vLLM on 32 chips with one 16-chip prefill worker and one 16-chip decode worker, KV offload to host memory and vLLM's router. GB300 NVL72 runs Dynamo vLLM on 8 to 32 GPUs.

The chart below shows total tokens per $1 TCO for both systems on Kimi K3:

As you can see, GB300 NVL72's frontier has no point between 106 tok/s/user (48 agents) and 210 tok/s/user (4 agents), so its values at 150 and 200 tok/s/user in the table below are interpolated across that gap.

Values are M tokens per $1 TCO, and ratios are TPU v7 / GB300 NVL72 on Dynamo vLLM.

At peak throughput GB300 NVL72 serves 20,143 tok/s/GPU with an 8-GPU aggregated config at 56 agents, or 31.4M tokens per $ on both bases, against 6,567 tok/s/chip for TPU v7 at 72 agents, or 19.5M tokens per $ on the external cost and 23.0M on the internal cost. That is 0.62x per dollar on the external cost, 0.73x on the internal cost, 0.33x per chip and 0.57x per MW. TPU v7's p90 time to first token is 4.7 s to 6.3 s between 32 and 72 agents. Its 72-agent point runs without MTP, and its other frontier points use MTP.

At 80 to 100 tok/s/user and on the external owning cost, TPU v7 delivers 2.15x to 2.26x the tokens per dollar of B200 and B300 and 0.91x to 1.07x that of the best MI355X line. On Google's internal cost the B200 and B300 ratios are 2.5x to 2.7x, and TPU v7 delivers 1.07x to 1.25x the tokens per dollar of the best MI355X line.

Some caveats

First, on Qwen3.5 TPU v7 serves from a single aggregated 4-chip host while GB300 NVL72 uses up to 32 disaggregated GPUs. Second, on Kimi K3 and below ~103 tok/s/user on Qwen3.5, the system that delivers more tokens per dollar changes between the external owning cost and Google's internal cost. Third, GB300 NVL72 has no frontier points on Qwen3.5 FP8 between 192 and 450 tok/s/user or on Kimi K3 between 106 and 210 tok/s/user, and matched values inside those gaps are interpolated. Fourth, AgentX does not evaluate model answer quality, so the accuracy difference between FP4 and FP8 is outside these results. Fifth, per-MW numbers use planning figures for all-in power and do not use power measured in these runs. Sixth, the engines differ: TPU v7 runs vLLM on Google's TPU backend, and GB300 NVL72 runs Dynamo SGLang on Qwen3.5 FP8 and FP4, Dynamo TensorRT-LLM on Qwen3.5 FP4 and Dynamo vLLM on Kimi K3. Seventh, the dashboard marks TPU7x results as an official preview that may change as validation and publication continue.

Configurations

What's next

TPU v7 has no multi-host or disaggregated Qwen3.5 AgentX configs yet, and its single-host curve stops at 32 agents. DeepSeek-V4-Pro, GLM-5.2 and MiniMax-M3 do not have TPU v7 AgentX results yet. TPU 8i, Google's first inference-specific TPU, adds native FP4 support, which Ironwood lacks.

All articles and posts are © SemiAnalysis. All rights reserved. The AGPL-3.0 license covering the application source code does not apply to article content.