Prefix caching
Also known as prompt caching, automatic prefix caching
In plain English
Prefix caching remembers the work for a repeated beginning, such as the same system prompt, so the model can skip that work next time.
Technical definition
Prefix caching reuses KV-cache state when multiple requests begin with the same token sequence.
Engineering details
A repeated system prompt, shared document, or common conversation prefix can reuse cached states. A cache hit can reduce prompt computation and time to first token.
Why it matters
Production workloads with repeated prefixes may outperform synthetic random-token benchmarks. The benefit depends on hit rate, cache capacity, eviction policy, and whether requests route to workers that hold the needed state.
How to read it in InferenceX
InferenceX disables prefix caching on random fixed-sequence datasets to isolate full prompt processing from cache policy, so treat those numbers as a no-hit baseline. AgentX inverts this: hit rate is a reported quantity there, shown per point alongside the offload tier it was served from, because reuse is the defining property of the workload.
Source material
See the concept in real benchmarks
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200