Gated DeltaNet
Also known as GDN, Gated Delta Net, delta rule attention
In plain English
Gated DeltaNet is a linear attention layer that keeps a fixed-size running state per request instead of a KV cache that grows with every token.
Technical definition
Gated DeltaNet is a recurrent linear attention mechanism in which, at each step, the running state is decayed by a gate, updated with a rank-one delta-rule correction, and projected by the query to produce the output.
Engineering details
Qwen3.5 interleaves GDN layers with grouped-query attention layers, making it a hybrid model with two kinds of state: GQA layers accumulate a growing KV history while GDN layers hold a fixed-size recurrent state per request. On TPU the recurrence is scheduled across the MXU, VPU, VMEM, and HBM. One optimization rearranges the output projection algebra so the MXU computes the decayed state times the query while the VPU builds the next state, removing the state update from the MXU dependency path; the current token’s contribution is a scalar dot product per head plus a small vector add. Other changes slice Q and K in the decode loop to cut register spills, store state in BF16 in HBM while computing in FP32, and fuse Conv1D with GDN into one kernel.
Why it matters
Fixed-size state makes long contexts cheap in memory but complicates prefix caching, because the recurrent state at the end of a cached prefix is normally overwritten as the request continues. Hybrid prefix caching gives GDN separate slots for reading a checkpoint and writing live state, aligned to KV block boundaries.
How to read it in InferenceX
Reported Ironwood gains on Qwen3.5 include 4.48% throughput at concurrency 512 from the algebra rearrangement, 11.3% from asynchronous state transfers, 15% on 1k8k from BF16 state storage, and kernel-level speedups of 1.41x decode, 1.60x prefill, and 2.14x mixed batches from GDN v3.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
AMD MI355X Qwen3.5 397B-A17B Inference: Up to 19x Throughput per GPU in 3 Months on SGLang FP8
From v0.5.8 (Feb) → v0.5.10rc0 (Apr) → v0.5.12 (May), three AITER kernel landings on MI355X plus a TP=8 → TP=2/TP=4 retune push Qwen3.5 8k/1k peak from 1.3k to 6.4k tok/s/GPU and extend the curve out to 75 tok/s/user
MI355X versus GB300 NVL72 Inference Performance: 20x Gap on Qwen3.5 SGLang
GatedDeltaNet, a 262k native context, and no AMD competition at all on the same engine