Sliding window attention
Also known as local attention, windowed attention
In plain English
Sliding window attention lets a layer look only at a recent span of tokens, so its cache stops growing once the window is full.
Technical definition
Sliding window attention restricts each query to a fixed span of preceding tokens, bounding the KV state a layer must retain.
Engineering details
Because the span is fixed, the cache for such a layer reaches a ceiling instead of growing with the conversation, and older entries fall out as the window advances. Models usually interleave these layers with full-attention layers, so long-range information still has a path through the network while most layers stay cheap.
Why it matters
The bounded cost comes with an allocator problem. Window pages turn over constantly while durable prefix pages sit still, and when both are drawn from one pool the transient allocation tends to evict the valuable one, so a long session can lose its expensive full-attention history to short-lived window state.
How to read it in InferenceX
These effects appear only on multi-turn traffic. A single short prompt never laps the window or contends for the pool, which is why window-aware eviction, offload, and branch handling show up as AgentX-driven engine work rather than as fixed-sequence results.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K