Serving
KV-cache affinity
Also known as cache affinity, session affinity, rank-aware routing
In plain English
KV-cache affinity sends related requests back to a worker or rank that already holds their prefix, avoiding unnecessary cache transfers or repeated prefill.
Technical definition
KV-cache affinity is a routing preference that preserves locality between a request’s reusable state and the worker or rank chosen to serve it.
Engineering details
A host can contain multiple data-parallel ranks with separate cache ownership. Routing to the correct host alone may therefore miss the useful prefix. A policy can track prefix locations or keep a session attached to its previous worker, while allowing reassignment when the worker is full or unavailable. Affinity is a preference, not a guarantee that state has survived eviction.
Why it matters
Locality can improve cache reuse but concentrate load on a subset of workers. A scheduler must balance the saved computation or transfer against queueing, capacity, and failover. Session affinity and general prefix-aware routing overlap without being identical policies.
How to read it in InferenceX
The TPU Qwen3.5 article attributes a decode cache-hit improvement to routing to the exact data-parallel rank. It also describes keeping sessions on a decode host unless the host is full, connecting cache locality to admission control.
See the concept in real benchmarks
All articlesThe Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
TPUv7 AgentX results across Muse Spark Open, GLM, Qwen3.5, and Kimi K3, with TorchTPU, megakernels, and llm-d disaggregation.
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200