KV cache
Also known as key-value cache, attention cache
In plain English
The KV cache is the model’s working memory for the current conversation. It keeps useful notes and avoids rereading everything for every new token.
Technical definition
The KV cache stores attention key and value states for tokens already processed, allowing each decode step to reuse them.
Engineering details
The cache grows with sequence length, batch size, layer count, and the number and width of stored attention heads. During decode it is repeatedly read from accelerator memory, so both capacity and bandwidth matter.
Why it matters
KV-cache pressure limits concurrency and long-context serving. Cache quantization, paged allocation, latent attention, prefix reuse, and disaggregated transfer systems all target its capacity or movement cost.
How to read it in InferenceX
InferenceX disables prefix caching for fixed-sequence comparisons on random data unless a recipe states otherwise, which keeps unrelated requests from receiving artificial cache hits. AgentX is the deliberate exception: its replayed sessions are built to reuse prefixes, so cache capacity, eviction policy, and offload are part of what the scenario measures.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200