Serving
KV cache eviction
Also known as prefix eviction, cache replacement
In plain English
KV cache eviction removes retained attention state from a cache tier so its space can be reused for other work.
Technical definition
KV cache eviction reclaims cached attention-state blocks according to a replacement policy when they are no longer retained in a memory tier.
Engineering details
A prefix cache can discard inactive blocks while protecting blocks still referenced by running requests. Recency, reuse, and prefix structure can guide replacement. Removing an accelerator copy does not necessarily destroy every copy: a lower-tier cache may retain the state for later reload.
Why it matters
Evicting a prefix that is reused soon adds transfer or recomputation work. A high cache occupancy therefore says little about whether useful state survives. Cache hit rate, reuse distance, and the workload’s branching pattern help explain the cost.
How to read it in InferenceX
AgentX tool pauses and subagent bursts create competing cache lifetimes. Distinguish eviction of inactive cached prefixes from preemption of an in-progress request, and distinguish discarding state from offloading it to another memory tier.
See the concept in real benchmarks
All articlesAgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure