N-gram embedding
Also known as multi-token embedding
In plain English
An n-gram embedding gives a short sequence of tokens a learned vector that the model can look up.
Technical definition
An n-gram embedding maps a local sequence of n tokens to learned vector features, extending the vocabulary of patterns beyond individual tokens.
Engineering details
Repeated names, code fragments, and common phrasing can activate these lookups. The article examines examples through Engram gate scores, but selects examples for interest rather than treating them as a representative ranking of memory use.
Why it matters
Local token patterns make lookup addresses available without waiting for hidden-state computation. That property supports prefetching and sparse parameter offload, although it does not prove that every retrieved feature is useful on every occurrence.
How to read it in InferenceX
The Engram gate scan does not establish how often each pattern appeared during training or how much table capacity a category occupies. A high gate score also does not measure access frequency, so it is insufficient for choosing which rows to cache.