CPU offloading
Also known as CPU offload, DRAM offloading, host memory offloading
In plain English
CPU offloading spills KV cache the accelerators cannot hold into the host machine DRAM, so long conversations resume from memory instead of being recomputed from scratch.
Technical definition
CPU offloading stores reusable KV cache blocks in host CPU DRAM instead of accelerator HBM and loads them back over the host link when a later request reuses that prefix.
Engineering details
In inference serving the term almost always means KV cache offloading to DRAM, distinct from the training-side practice of parking weights or optimizer state on the CPU. Engines reach DRAM through connectors such as the vLLM CPU offloading connectors, LMCache, SGLang HiCache, Mooncake Store, and Dynamo KVBM. The pool is usually write-through, so it pays off when host DRAM for offload is roughly 1.5 to 3 times HBM KV capacity, and transfer efficiency decides the rest: AMD vLLM could not batch GPU-to-CPU copies before hipMemcpyBatchAsync landed in ROCm 7.14, which made its CPU offload path far less useful than the same feature on NVIDIA.
Why it matters
DRAM offloading decides how many concurrent agent sessions stay resumable once their combined KV working set exceeds HBM. It is not free capacity: at high concurrency, heavy reliance on the DRAM tier adds reload traffic that can push interactivity below acceptable levels, so the useful question is when the tier helps rather than whether it exists.
How to read it in InferenceX
AgentX treats CPU KV offloading as an allowed, optional optimization. Offload DRAM must scale with the fraction of GPUs used, with a 3 TB cap for systems without standardized DRAM configurations. Points that used offload are ringed with a dashed halo, and the point detail view reports the offload backend plus HBM and CPU cache hit rates.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set
50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from