Multi-head latent attention
In plain English
MLA compresses the model’s notes about earlier tokens so long conversations use less memory and are cheaper to continue.
Technical definition
Multi-head latent attention compresses attention key and value state into a lower-dimensional latent representation to reduce KV-cache size and memory traffic.
Engineering details
Instead of storing full per-head keys and values for every prior token, MLA stores compressed state and reconstructs or consumes the needed representations through model-specific projections. Implementations require specialized attention kernels.
Why it matters
Reducing KV-cache bytes increases feasible context length and concurrency and can lower decode bandwidth pressure. Kernel shape support and tensor-parallel layout can still create large performance differences.
How to read it in InferenceX
Several DeepSeek-derived models in InferenceX use MLA. Articles track fixes where an attention backend handled one heads-per-rank shape efficiently but failed or fell back on another.
Source material
See the concept in real benchmarks
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K
MI355X DeepSeek-V4-Pro on SGLang: 110.5x Throughput per GPU in 26 Days
The amd/deepseek_v4 side branch shipped TileLang attention indexer, Triton sparse MLA, fused RoPE/Hadamard, FlyDSL MoE, and FP4 weights across 31 performance optimizations PRs — lifting first-light 20 tok/s/GPU at 2.4 tok/s/user into 2,256 tok/s/GPU at 9.4 tok/s/user on 8K/1K, with both throughput and interactivity climbing together
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool