KV cache manager
Also known as Mooncake, LMCache, HiCache
In plain English
A KV cache manager is the component that stores attention state outside the chips and decides what to keep, evict, and fetch back.
Technical definition
A KV cache manager is a pluggable layer beneath an inference engine that stores reusable KV blocks across memory tiers and manages their placement, eviction, and transfer.
Engineering details
Engines expose a connector interface, so managers such as Mooncake Store, LMCache, and SGLang HiCache can serve different runtimes. The manager keys blocks by prefix hash and places them in host DRAM, local NVMe, or a remote backend, while a separate transfer engine such as Mooncake Transfer Engine or NIXL performs the byte movement. Several paths can coexist inside one engine.
Why it matters
Once a workload reuses prefixes heavily, correctness and accounting in this layer matter as much as kernel speed. Hybrid-attention models make it harder still, because a model carrying several cache groups with different shapes and lifetimes cannot be described by a connector that assumes one uniform block geometry.
How to read it in InferenceX
InferenceX records the KV offload backend as run metadata and shows it in the AgentX point detail view. Framework labels name the combination rather than the engine alone, so a recipe reads as Mooncake ATOMesh or MoRI SGLang instead of just its engine.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM