CPU offloading
Also known as CPU offload, DRAM offloading, host memory offloading
In plain English
CPU offloading keeps model data or reusable attention state in host DRAM, freeing accelerator memory while adding host-memory access costs.
Technical definition
CPU offloading stores model parameters or KV cache in host CPU DRAM instead of accelerator HBM. The serving implementation determines whether data is explicitly copied or accessed directly by the GPU.
Engineering details
KV cache offloading preserves request-specific attention state through systems such as LMCache, SGLang HiCache, Mooncake Store, and Dynamo KVBM. Parameter offloading instead places learned model data in DRAM. The Engram article describes a GPU kernel selecting and dequantizing sparse embedding rows directly from pinned host memory through UVA, while retaining full decode graphs. These paths have different access patterns and should be named separately.
Why it matters
DRAM offloading decides how many concurrent agent sessions stay resumable once their combined KV working set exceeds HBM. It is not free capacity: at high concurrency, heavy reliance on the DRAM tier adds reload traffic that can push interactivity below acceptable levels, so the useful question is when the tier helps rather than whether it exists.
How to read it in InferenceX
AgentX treats CPU KV offloading as an allowed, optional optimization. Its KV-offload markers and cache hit rates describe that path, not Engram parameter placement. The Engram article reports a B300 configuration moving from TP4 to TP2 with DRAM offload and improving its Pareto curve by up to 1.6x; this is a measured configuration result, not a universal DRAM speedup.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set
50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from
Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
New Model Architecture Implications for TAM of DRAM/NVMe, DeepSeek V4.1 Flash, AgentX, InferenceX, NVMe experiments