Parameter offloading
Also known as model weight offloading, Engram offloading
In plain English
Parameter offloading keeps learned model data outside accelerator memory and fetches the portions needed for computation.
Technical definition
Parameter offloading stores learned model parameters in a lower memory tier, such as host DRAM or SSD, while the accelerator accesses the data required for inference.
Engineering details
Engram is suited to sparse row retrieval because row addresses follow token IDs. This differs from KV cache offloading, which moves attention state generated for particular requests. The two mechanisms can compete for, or free capacity within, the same memory hierarchy.
Why it matters
Moving parameters out of HBM can leave more room for KV cache or allow fewer GPUs per replica. Those benefits must outweigh access costs and depend on the kernel, interconnect, memory allocation, and workload.
How to read it in InferenceX
The article reports up to a 1.6x Pareto improvement when its B300 DRAM-offload configuration moves from TP4 to TP2. Its unoptimized B200 SSD path performs worse than DRAM, showing that cheaper storage alone does not establish a lower cost per token.