KV cache offload
Also known as CPU offload, KV offloading
In plain English
KV cache offload parks attention state the chips cannot hold in host memory, so a long session can be resumed instead of recomputed.
Technical definition
KV cache offload moves KV blocks out of accelerator memory into a slower tier, usually host DRAM, and loads them back when a later request reuses that prefix.
Engineering details
Offload is usually a write-through cache: a prefix written to the HBM cache is also written to the slower tier, so it helps most when the offload pool is roughly one and a half to three times HBM capacity. Reloading a long prefix beats recomputing it by a wide margin at agentic context lengths, but the arithmetic reverses for short prompts, where the transfer costs more than the prefill it avoids.
Why it matters
Long agentic sessions exceed HBM KV capacity well before they exceed a plausible DRAM budget, so offload decides how many concurrent conversations stay resumable. It also shifts the bottleneck: once prefixes survive, store and load paths, transfer batching, and index bookkeeping become the costs worth optimizing.
How to read it in InferenceX
InferenceX rings every point that used KV offload with a dashed halo, whether or not it is Pareto optimal, and the point detail view names the offload type and engine alongside the chip and CPU cache hit rates. Offload is an allowed but optional optimization, so a single curve can mix points with and without it.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set
50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from