vLLM

Highly concurrent agentic workloads require offload, and AgentX drove the work that made CPU KV offload usable for hybrid models rather than only for uniform full-attention models. The distinction matters: a uniform model has one KV layout per token, so a connector can describe what to save with a single block geometry. A hybrid model carries several cache groups at once, each with a different shape and a different lifetime, and a connector that assumes one uniform layout cannot express which group a given block belongs to. Offload was therefore unavailable for exactly the models whose long sessions needed it most.

The general SimpleCPU connector came first, was enabled on ROCm, and was then extended to DeepSeek-V4 hybrid attention, which reported 81.7% higher output throughput and 46.6% lower mean end-to-end latency against recomputing the prefix once it no longer fits in HBM. Mooncake gained the equivalent hybrid-memory allocation support.

Profiling realistic workloads showed that once offload worked at all, the cost moved to the store path, which was writing too much and too often. Three rules narrowed it.

A store is now skipped while an identical transfer is already in flight, so concurrent sessions sharing a prefix pay for it once rather than once each. A store covers only newly generated KV ranges, so a session that extends its history writes the delta instead of rewriting the whole prefix every turn. And a store no longer depends on whether the same blocks still sit in HBM, so work already scheduled is not discarded when an eviction lands underneath it.

The load path was tuned separately, because lookups happen on every scheduling decision rather than only when data actually moves. Making lookups asynchronous in the scheduler path keeps the connector off the step’s critical path, so a step no longer waits on CPU-side cache queries before it can admit work.

Compact zero-copy lookup keys, parallel receive-side loading, and prebuilt Mooncake key strings then removed the CPU and transport overhead that remained.

Long-lived hybrid state forced correctness and accounting fixes that fixed-shape requests rarely reach. vLLM now emits cache events per hybrid cache group, strides distributed-context stores correctly, and computes lookup prefixes correctly under distributed context and prefill. A related context-parallel accounting change aligns cache ownership with sharded token ranges.

Speculative state is now propagated across merged Mooncake groups and through the SimpleCPU coordinator, which prevents the repeated-turn cache from silently losing EAGLE state.

On the ROCm side the recent work continues below the cache layer, where the remaining cost is per-layer rather than per-request. Once the prefix survives and arrives on time, what is left is the decode step itself, and a decode step that runs thousands of times per session pays for every avoidable copy and every mismatched kernel. Three open changes attack that layer.

A Kimi-K3 change writes KDA decode results directly into the layer output buffer, removing one device copy per KDA layer; the saving is small in isolation and repeated for every layer of every decoded token. A second change selects an AITER sparse-MLA decode kernel in place of the generic path, and reported 5.22% higher AgentX output throughput with substantially lower inter-token latency.

The third is a useful illustration of how much the measurement shape matters. A companion change routes full-graph attention projections through tuned AITER GEMMs and reached a 2.3% gain on fixed sequences at low concurrency. A kernel-level change can show a clean gain on uniform shapes and then be swamped, on an agentic trace, by the cache and scheduling variance the trace introduces.