Linear attention
Also known as GatedDeltaNet, recurrent attention, constant-state attention
In plain English
Linear attention keeps a fixed-size running summary instead of every past token, so its memory does not grow as the conversation does.
Technical definition
Linear attention replaces the growing key and value cache with a recurrent state of constant size that is updated as tokens arrive.
Engineering details
Standard attention stores state proportional to sequence length and rereads it every step. A linear or gated recurrent layer carries a fixed-size state instead, trading exact recall of every position for bounded memory. Architectures such as GatedDeltaNet apply this on a fraction of layers, leaving full attention elsewhere to preserve precise long-range lookup.
Why it matters
For long context the storage saving is substantial, but the state changes what caching means. It cannot be reconstructed from surrounding tokens the way a window tail can, so if it is dropped the only way back is replaying the sequence, and reuse requires an explicit checkpoint mechanism rather than ordinary block reuse.
How to read it in InferenceX
InferenceX serves models using these layers, and engine support for checkpointing and transferring recurrent state is part of the recipe. A model can be day-zero servable and still lack reuse of that state, which shows up as unexpectedly high prefill cost on repeated turns.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
AMD MI355X Qwen3.5 397B-A17B Inference: Up to 19x Throughput per GPU in 3 Months on SGLang FP8
From v0.5.8 (Feb) → v0.5.10rc0 (Apr) → v0.5.12 (May), three AITER kernel landings on MI355X plus a TP=8 → TP=2/TP=4 retune push Qwen3.5 8k/1k peak from 1.3k to 6.4k tok/s/GPU and extend the curve out to 75 tok/s/user
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
Qwen3.5 397B on AgentX: A 20x SGLang Gap at 90 tok/s/user
GatedDeltaNet, a 262k native context, and no AMD competition at all on the same engine