KV head replication
Also known as KV head replication, replicated KV heads, KV head duplication
In plain English
KV head replication copies the same key-value heads onto several tensor-parallel ranks when a model has fewer KV heads than the devices it is split across.
Technical definition
KV head replication is the tensor-parallel strategy of duplicating key and value heads across ranks when the tensor-parallel degree exceeds the number of KV heads in a grouped-query attention layer, so each rank has the KV data its local query heads need without a communication step.
Engineering details
Qwen3.5 has 32 query heads and 2 shared KV heads. With TP8 each device gets 4 query heads, but 2 KV heads do not divide across 8 devices; each KV head is shared by 16 query heads, so four devices need the same KV data. Replicating both KV heads on every rank avoids an all-to-all exchange of KV tensors at the cost of duplicating KV cache memory on each rank. vLLM already supported this, and the TPU backend needed a compatibility fix to activate the existing behavior.
Why it matters
The alternative for high concurrency is DP attention: each device handles a different subset of requests and holds both KV heads for those requests, so KV memory is not duplicated across ranks while the much larger expert weights stay sharded with expert parallelism. Ironwood serving uses TP8 attention at low concurrency and DP8 plus EP8 at high concurrency.
How to read it in InferenceX
Activating replication removed unnecessary All-to-All communication in the Qwen3.5 397B TorchTPU bring-up. The low-concurrency tuning round that switched to TP8 attention plus expert parallelism gave 1k1k gains of 22.9% at concurrency four and 18.1% at eight.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200