KV cache DRAM P2P pooling
Also known as P2P pooling, DRAM pooling, pooled KV storage, shared KV pool
In plain English
P2P pooling joins the host memory of many servers into one logical KV cache store, so an accelerator on any server can pull cached context from any other.
Technical definition
KV cache DRAM P2P pooling is an offload architecture in which the host DRAM contributed by every node in a cluster is aggregated into a single logical memory pool, letting any accelerator read KV cache written by any other node over the network.
Engineering details
Single-node offload moves KV blocks from HBM to the local host DRAM when HBM fills. Pooling goes further: Mooncake Store unifies the DRAM on every TPU host into one shared pool, and can pool NVMe from multiple servers as well or back onto distributed filesystems like WEKA and VAST. A request routed to a different server than the one that computed its prefix can still hit in cache. Google is externalizing its native TPU offload stack through TPU-Sync and supporting Mooncake Store, likely built on TPU-Sync primitives.
Why it matters
Pooling raises the achievable prefix cache hit rate for agentic workloads, where sessions run hundreds of turns and sub-agents burst with fresh context. Without it, hit rate is capped by what one node can hold and by how well the router keeps a session on one machine.
How to read it in InferenceX
TPU-Sync DRAM offload and Mooncake Store pooling are listed as next steps in the TPU InferenceX preview, ahead of AgentX TPU results. The AgentX articles on NVIDIA and AMD already show that KV working set size and offload capacity decide agentic cost per token.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure