Hybrid prefix caching
Also known as hybrid prefix caching, GDN checkpoint caching, aligned checkpoint mode
In plain English
Hybrid prefix caching lets a model with both attention and linear-attention layers reuse a cached prompt by saving the recurrent state alongside the KV blocks.
Technical definition
Hybrid prefix caching is the serving technique that stores, for a shared prompt prefix, both the KV blocks of the attention layers and a checkpoint of the recurrent state of the linear-attention layers, so a new request can resume from the prefix without recomputing it.
Engineering details
For a plain attention model, caching a prefix means keeping its KV blocks. For a hybrid, the GDN recurrent state at the end of the prefix must also survive, but the live state is normally overwritten as the request continues. The Ironwood implementation gives GDN separate slots for reading a checkpoint and writing the live state, derives state addresses from the block table that already locates KV blocks, and takes checkpoints at an aligned cache granularity so a saved state always lines up with a KV block boundary. This mode needs a full checkpoint pool rather than the compact per-request allocation, trading HBM for reuse.
Why it matters
Agentic and multi-turn workloads reuse long system prompts and conversation histories, and hybrid models cannot benefit unless the recurrent state is cached too. The trade is explicit: prefix caching for hybrids costs the HBM that compact allocation had saved, so the mode pays off only when prefixes are actually shared.
How to read it in InferenceX
The Qwen3.5 397B numbers in the TPU InferenceX preview are random-input 8k1k runs with nothing shared between requests, so they do not exercise this path. Hybrid prefix caching with DP support is the groundwork for the AgentX TPU results Google customers have asked for.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure