Prefix caching
Also known as prompt caching, automatic prefix caching
In plain English
Prefix caching remembers the work for a repeated beginning, such as the same system prompt, so the model can skip that work next time.
Technical definition
Prefix caching reuses KV-cache state when multiple requests begin with the same token sequence.
Engineering details
A repeated system prompt, shared document, or common conversation prefix can reuse cached states. In a growing agent session, earlier outputs become part of later inputs, increasing potential prefix reuse. Actual hits still depend on the state remaining available and reachable; eviction or fresh subagent contexts can require new prefill.
Why it matters
Production workloads with repeated prefixes may outperform synthetic random-token benchmarks. The benefit depends on hit rate, cache capacity, eviction policy, and whether requests route to workers that hold the needed state.
How to read it in InferenceX
InferenceX disables prefix caching on random fixed-sequence datasets to isolate full prompt processing from cache policy, so treat those numbers as a no-hit baseline. AgentX inverts this: hit rate is a reported quantity there, shown per point alongside the offload tier it was served from, because reuse is the defining property of the workload.
Source material
See the concept in real benchmarks
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Rubin NVL72 Agentic Inference: 67x better Performance per Dollar
Jensen Sandbagging Performance Again, 2x more Annual Profit Per GigaWatt, The More you Buy, The More you Earn, AgentX, InferenceX, Extreme Co-Design