TPU-Sync
Also known as TPU-raiden, tpu-sync, TPU KV transfer library
In plain English
TPU-Sync is Google’s library for moving KV cache between TPUs and out to host memory, the plumbing that disaggregated serving and offloading need.
Technical definition
TPU-Sync, formerly TPU-raiden, is Google’s open-sourced disaggregated KV-cache transfer library for TPUs that performs zero-copy transfers by extracting native PJRTBuffer hardware descriptors.
Engineering details
Prefill-decode disaggregation needs a fast path to move a finished prefill’s KV cache from one pool of chips to another, and KV-cache offloading needs a path from HBM to host DRAM and back. TPU-Sync supplies both. It works natively with JAX and with the TorchTPU stack, and it supports native TPU KV-cache DRAM offloading for large models and medium-to-large batches where HBM cannot hold every user’s cache. Google has run disaggregation internally for Gemini for years; externalizing it began only a couple of months before the Official Preview, alongside TPU support in llm-d.
Why it matters
Without a transfer library, TPU external serving is limited to aggregated mode, which is where the current results sit and where GB200 and GB300 NVL72 disaggregated serving currently leads on part of the curve. SemiAnalysis expects Mooncake Store support on TPU to be built on TPU-Sync primitives.
How to read it in InferenceX
The InferenceX Official Preview compares aggregated TPUv7 against aggregated B200 and B300. A TPUv7 disaggregated versus GB200 and GB300 NVL72 disaggregated comparison is planned for a follow-up once TPU-Sync-based disaggregation is optimized in the external stack.