KV-aware routing
Also known as cache-aware routing, session affinity
In plain English
KV-aware routing sends a request to the worker that already holds its conversation state, instead of to whichever worker is least busy.
Technical definition
KV-aware routing selects a worker using where cached prefix state already resides, rather than on queue depth or load alone.
Engineering details
A request carrying no reusable history can go anywhere, and load balancing is the only question worth asking. A request carrying megabytes of cached prefix is different: sending it to an idle worker that lacks that prefix pays for the whole prompt again. Routers therefore track cache events, hash sessions to consistent workers, and keep data-parallel ranks sticky to the sessions whose state they own.
Why it matters
Under data-parallel attention each rank owns a private slice of the cache pool, so a long session landing on the wrong rank recomputes everything and the measured hit rate collapses far below its theoretical ceiling. Affinity alone is not enough either, since unchecked stickiness concentrates load on one hot worker, so cache balance has to enter the routing score.
How to read it in InferenceX
Routing sits outside the engine, so InferenceX treats it as part of the recipe: labels such as Dynamo vLLM, llm-d vLLM, and Mooncake ATOMesh name the orchestration layer as well as the runtime. Its cost scales with the number and length of live prefixes rather than with tokens generated, which is why it can become the bottleneck on agentic traffic once kernels improve.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure