Mooncake Store
Also known as Mooncake, Mooncake KV store, P2P KV pooling
In plain English
Mooncake Store pools the host memory and NVMe drives across many servers into one shared KV-cache store that any accelerator in the cluster can read.
Technical definition
Mooncake Store is the open-source distributed KV-cache storage layer from the Mooncake project that aggregates DRAM and NVMe across serving nodes into a single logical pool with peer-to-peer access.
Engineering details
With P2P pooling, the KV-cache storage on each host is presented as one logical memory pool, so an accelerator on any server can fetch cache written by any other server instead of only its own host. Mooncake Store also pools NVMe across servers and supports distributed filesystem backends such as WEKA and VAST. It is used with vLLM and SGLang on GPUs as the industry-standard offloading store, and Google is adding TPU support along with the DRAM P2P pooling mode, likely implemented on TPU-Sync primitives.
Why it matters
Agentic and multi-turn workloads reuse long prefixes across many requests and across sub-agent bursts. A cluster-wide pool raises the KV-cache hit rate well beyond what a single host’s DRAM allows, which is what makes high-concurrency agent serving economical once HBM is exhausted.
How to read it in InferenceX
AgentX results on GPUs already depend on offloading behavior, and the planned AgentX TPU results will exercise Mooncake Store and TPU-Sync on Ironwood. The InferenceX Official Preview lists Mooncake support as one of the next externalization steps after disaggregation.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance