NVMe offloading
Also known as NVMe offload, SSD offloading, flash KV cache offload
In plain English
NVMe offloading stores model data or attention state on SSDs, trading cheaper capacity for a more expensive access path.
Technical definition
NVMe offloading places model parameters or reusable KV cache on NVMe SSDs below the accelerator and host-memory tiers. Parameter lookup and KV-prefix restoration are distinct uses of the same storage tier.
Engineering details
KV offloading can preserve prefixes after DRAM eviction. Engram offloading instead retrieves learned embedding rows. The article tests a memory-mapped SSD-backed Engram table whose pages may remain in the filesystem cache. Its unoptimized path copies row IDs to the CPU, deduplicates and gathers rows, then transfers them to the GPU. A warm page cache eliminates storage reads but not this coordination or copying.
Why it matters
NVMe offloading effectively lengthens cache lifetime for long-lived agent sessions, which matters as agents wait on tools, humans, or CI for minutes at a time. It is workload dependent: a high-concurrency deployment where DRAM offloading already degrades latency will not be rescued by an even slower tier, because the bottleneck is reload bandwidth rather than capacity.
How to read it in InferenceX
The Engram article reports about 121 million total tokens per dollar for DRAM versus 52 million for SSD near 125 tokens/s/user on its B200 configurations. GDS was not enabled, and the experiment measures the whole serving path rather than isolated SSD latency. This parameter-offload experiment does not establish support for NVMe KV offloading in AgentX.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
New Model Architecture Implications for TAM of DRAM/NVMe, DeepSeek V4.1 Flash, AgentX, InferenceX, NVMe experiments