Sparse embedding lookup
In plain English
A sparse lookup reads a few selected embedding rows rather than reading the entire parameter table.
Technical definition
Sparse embedding lookup retrieves only the rows selected by an input-dependent index set, so accessed data per token can be small relative to total table capacity.
Engineering details
The Engram configuration in the article requests 24 rows at each of two layers. It reports about 12.4 KiB per processed token position across the model, or 3.1 KiB per GPU when split across four GPUs.
Why it matters
Sparse traffic makes offloading plausible but introduces irregular memory access. Row selection, dequantization, transfer overhead, and reuse determine whether a lower memory tier can supply data without delaying the rest of the model.
How to read it in InferenceX
Moving the Engram table into HBM accelerates its sparse lookup without directly accelerating decoder computation or communication. The article therefore evaluates full AgentX serving curves instead of assuming a faster lookup must produce a proportionate end-to-end gain.