Optimizations for Agentic Workloads

The work collected on the project pages addresses long-context cost: a prefix that has to survive, a hybrid cache that has to stay correct, a transfer that has to keep up. The changes here are different in kind. They are day-zero enablement and correctness bugs, and they break a request exactly as badly as a million-token session does.

MiniMax-M3 tested whether the ROCm work compounds into day-zero readiness, and the Advancing AI writeup draws the comparison directly: AMD’s first public disaggregated recipe, MI355X FP4, reached InferenceX in January, months behind NVIDIA, while M3 FP4 disaggregation landed on day zero. That is a marked improvement on the DeepSeek-R1 period, when parity took months. Three vLLM fixes sat on the day-zero path, and each was a correctness failure rather than a performance one.

Disaggregation was blocked first. NixlConnector’s handshake asserted that the SPLIT-region block_len scales with the prefill-to-decode TP ratio, but block_len follows per-rank KV heads. M3 has 4 KV heads, so a TP4 prefill paired with a TP8 decode is GQA-capped to one head per rank on both sides, and the two lengths are equal where the assertion demanded a factor of two. The handshake was rejected, no KV moved, decode regenerated everything from scratch, and gsm8k scored 0. Validating against the actual head ratio fixed it.

The other two were platform splits. M3’s sparse-attention backend read the byte-backed FP8 cache as float8_e4m3fn for every E4M3 configuration, but gfx942’s platform dtype is e4m3fnuz. The two encodings differ, so K and V were altered before the kernels consumed them, and the prefill and decode wrappers had also omitted the FNUZ types from their FP8 checks. Using the platform dtype for the cache view fixed both halves. Separately, M3 ships as separate NVIDIA and AMD model files and only the NVIDIA one implemented the EAGLE3 interface, so speculative decoding aborted at engine init on ROCm with a model-does-not-support error. Bringing the AMD model to parity restored it, with MI355X gsm8k matching both the non-EAGLE3 MI355X run and B200.

The TensorRT-LLM page covers the M3 work that is long-context specific: descriptor explosion in disaggregated KV transfer, context graph capture, sparse block strides, autotuner candidates, and the corrupt split-K MoE tactics that had to be removed from the pool.

The local AgentX matrix combines session-aware or KV-aware routing, long and variable conversation histories, MTP, hybrid attention, aggregated and disaggregated serving, and concurrency sweeps that cross the HBM capacity cliff. It includes GPU-resident comparisons as well as CPU DRAM offload through vLLM SimpleCPU, Mooncake, LMCache, and SGLang HiCache. That combination is what activates the upstream work above.

The old fixed-sequence matrix usually creates one prompt, performs one prefill, decodes one fixed continuation, and discards the request. It therefore does not measure cache survival across turns, repeated tokenization, session affinity, cache-event traffic, offload churn, transfer progress during scheduler stalls, or long-lived ownership bookkeeping.

The allowed optimization policy treats CPU KV offload as optional. A vendor may use vLLM connectors, LMCache, SGLang HiCache, Mooncake, Dynamo KVBM, or another CPU DRAM connector — or disable offload when the resulting latency and throughput point is better. NVMe offload is deferred. CPU DRAM must scale with the fraction of GPUs used, including the 3 TB cap for systems without standardized DRAM. Systems with standardized DRAM have no hard cap but keep the same proportionality rule; the local generator currently applies the 3 TB cap to every runner, so it does not yet implement the standardized-DRAM exception.

The net new optimization surface is not simply longer attention. It is the preservation, movement, routing, reconstruction, and repeated processing of a growing session state. AgentX made those costs large enough to drive generic upstream changes across vLLM, SGLang, TensorRT-LLM, ATOM, AITER, Dynamo, and LMCache. Direct searches of NIXL and Mooncake did not identify additional AgentX-tagged runtime PRs beyond those listed here, so their effects remain represented through the engine connector changes.