Prefix cache hit rate
Also known as cache hit rate, KV reuse rate
In plain English
The hit rate is the share of prompt tokens served from cache instead of being recomputed, which on long sessions is most of the prompt.
Technical definition
Prefix cache hit rate is the fraction of input tokens satisfied from cached KV state rather than recomputed during prefill.
Engineering details
A hit rate is only meaningful next to the tier that produced it, since a token served from accelerator memory and one fetched back from host memory cost very differently. It also depends on more than capacity: eviction can discard a prefix that is still wanted, and routing can send a request to a worker that never held it.
Why it matters
On multi-turn traffic the hit rate largely determines prefill cost, because each turn resends the whole conversation plus a little more. Once reuse is high, the remaining prefill work is dominated by genuinely new tokens, and the bottleneck moves from computation to cache management.
How to read it in InferenceX
The AgentX point view reports hit rate over time, separated by cache tier, alongside the prompt token source breakdown. A run that reports high aggregate throughput on a low hit rate is doing far more prefill work than a well-cached deployment would.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set
50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from