When normalized by total cost of ownership, B300 vLLM and B200 vLLM aggregated performance on DeepSeek V4 Pro is quite similar. The main difference is that B300 can squeeze out extra throughput, given its 50% increase in HBM capacity over B200.

The interesting part is not the summary number, it is what the AgentX server telemetry shows underneath it.

43 million tokens against 22 million

Under the load of 384 concurrent agentic traces, B300 vLLM DEP8 with 3TB of DRAM using vLLM simple offloading achieved a 91% HBM cache hit rate, with an additional 1.36% DRAM cache hit rate. The reason is that the HBM KV cache working set size is approximately 43 million tokens with this configuration, and the load barely exceeds that number of tokens in flight at any given time.

With B200 at concurrency 196, all other parameters unchanged, the picture inverts. The HBM cache hit rate falls to 73% and the deployment leans much harder on DRAM, with an offload cache hit rate of nearly 20%. The HBM KV cache working set size here is 22 million tokens, roughly half that of B300.

Why the offload tier needs headroom

DRAM KV offloading is typically implemented as a write-through cache, meaning every prefix written to the HBM cache is also written to the DRAM cache. It is therefore most effective when the amount of DRAM available for offloading is significantly bigger than HBM KV cache capacity, by a multiple of roughly 1.5 to 3.

That property is what makes the extra HBM on B300 compound rather than merely add. A larger resident working set keeps more of the session in the fast tier, which leaves the slower tier serving a genuine remainder rather than absorbing a constant stream of evictions.

The H200 counterexample

H200 SGLang FP8 is able to serve DeepSeek V4 at low concurrency, and is even competitive with B200 and MI355X SGLang from a performance per dollar standpoint. It cannot compete with the newer SKUs in high throughput scenarios, because it lacks the HBM to do so.

The failure mode is instructive: the reliance on DRAM KV offloading at higher concurrencies leads to unreasonable latency as the number of users scales. Offload buys capacity, but it is not free, and a deployment that depends on it to cover a structural memory shortfall pays for that at the tail.

These results are one slice of AgentX 1.0. The full analysis, the replay methodology, and the 70+ upstream PRs the benchmark drove are in AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?. Every point is explorable on the free dashboard.