Data-parallel attention
Also known as DP attention, attention data parallelism
In plain English
DP attention gives each rank its own slice of the attention work and its own cache, instead of every rank holding a copy of the same thing.
Technical definition
Data-parallel attention replicates attention computation per rank over disjoint sets of sequences, so each rank owns a private share of the KV cache while experts remain shared.
Engineering details
Tensor-parallel attention splits heads and ends up replicating KV state across ranks, which wastes capacity when heads are few. DP attention avoids that duplication by assigning whole sequences to ranks. The ranks still participate together in the MoE collectives, so attention is local while expert dispatch stays global.
Why it matters
Because each rank owns a private slice of the pool, placement becomes correctness-adjacent for performance: a long session routed to a rank that does not hold its prefix recomputes everything. Measured hit rates can then land far below the theoretical ceiling for reasons that have nothing to do with cache size.
How to read it in InferenceX
InferenceX shows DP attention in the parallelism section of point tooltips. Whether it helps is model dependent, and configurations without it sometimes dominate the frontier when cache locality turns into a routing constraint.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4
MiniMax M3 on AgentX: B300 TRT-LLM TP2 Owns the Crown
NVIDIA sweeps the 432B model, and the missing DP-attention points explain why cache locality became a routing constraint