Recurrent state
Also known as recurrent state, GDN state, linear attention state, SSM state
In plain English
Recurrent state is the fixed-size memory a linear-attention or SSM layer keeps per request, unlike a KV cache that grows with every token.
Technical definition
Recurrent state is the fixed-size per-request tensor that a linear attention layer such as Gated DeltaNet updates at each step, replacing the growing per-token key-value history that softmax attention layers store.
Compact recurrent-state allocation, Qwen3.5 397B
About 76 GiB HBM reclaimed, attention block pool +71%, 1k8k throughput +18% at concurrency 64
Engineering details
In a hybrid model like Qwen3.5, GQA layers accumulate a KV history that grows with context while GDN layers hold a fixed-size state per request that is decayed, updated with a rank-one term, and read out each token. Serving engines must allocate slots for this state, decide its storage precision, and handle checkpoints for prefix caching, because the live state is overwritten as soon as the request continues. On Ironwood the state was initially allocated as num_blocks slots per layer group, which wasted HBM, then compacted to roughly one slot per active request.
Why it matters
Recurrent state changes the memory budget: it does not grow with context, which is the whole point of hybrid architectures, but it must be stored somewhere and moved between VMEM and HBM every step. Storing it in BF16 while computing in FP32 halves its footprint. Prefix caching for hybrids needs separate read and write slots so a cached checkpoint survives.
How to read it in InferenceX
Compact allocation reclaimed about 76 GiB of HBM and expanded the attention block pool 71%, improving Qwen3.5 397B 1k8k throughput 18% at concurrency 64. BF16 state storage gave a further 15% on 1k8k at concurrency 512.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200