NVMe offloading
Also known as NVMe offload, SSD offloading, flash KV cache offload
In plain English
NVMe offloading extends the KV cache one tier further, onto local SSDs, so prefixes that no longer fit in GPU or CPU memory can still be reloaded later.
Technical definition
NVMe offloading stores reusable KV cache blocks on NVMe SSDs beneath the HBM and host DRAM tiers, trading slower reloads for a much larger retrievable KV working set.
Engineering details
Each step down the memory hierarchy multiplies capacity and divides bandwidth, so the SSD tier only pays off when reloading a long prefix still beats recomputing it. KV cache managers such as LMCache and Mooncake Store already support local NVMe backends alongside DRAM and remote storage. The tier helps most when the reuse working set exceeds any plausible DRAM budget or when sessions return after idle gaps long enough that DRAM eviction has already discarded them.
Why it matters
NVMe offloading effectively lengthens cache lifetime for long-lived agent sessions, which matters as agents wait on tools, humans, or CI for minutes at a time. It is workload dependent: a high-concurrency deployment where DRAM offloading already degrades latency will not be rescued by an even slower tier, because the bottleneck is reload bandwidth rather than capacity.
How to read it in InferenceX
AgentX v1 measures HBM and DRAM tiers and defers NVMe offloading, with SSD/NVMe KV offloading planned as a fast follow to grow the working set beyond DRAM. The current 5 minute idle cap on replayed streams may rise alongside it so that longer cache lifetimes become measurable.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure