Embedding prefetch
Also known as Engram prefetch, row prefetching
In plain English
Embedding prefetch starts retrieving the rows a later model layer will need while earlier layers are still computing.
Technical definition
Embedding prefetch is the early retrieval of selected embedding rows so that memory access can overlap computation preceding their use.
Engineering details
Engram row addresses depend on token IDs rather than hidden states. Once those IDs are available, the runtime can determine the required rows without waiting for the intervening model layers to finish. The useful overlap window ends when the consuming layer needs the data.
Why it matters
Prefetch can hide some latency but does not remove transferred bytes or bandwidth limits. A serving system must still allocate buffers, coordinate completion, and avoid consuming data before retrieval has finished.
How to read it in InferenceX
The article attributes the competitiveness of its DRAM path to optimizations including asynchronous execution and overlap. It does not isolate a universal prefetch speedup, so compare the full configuration and its concurrency range rather than extrapolating from the mechanism.