TensorRT-LLM

The MiniMax-M3 work focuses on disaggregated KV movement, where the failure is one of granularity. When prefill and decode do not agree on head layout, the KV for one logical request stops being a few large contiguous regions and becomes thousands of small strided pieces, each of which turns into its own transfer descriptor. The bytes moved are unchanged; the per-descriptor overhead is what explodes, and it explodes worst on exactly the long prompts that matter.

Corrected multi-pool mapping and a chunked NIXL bounce path coalesce those pieces through a bounded reusable arena, trading an extra staging copy for orders of magnitude fewer descriptors. The AgentX diagnostic reduced request-critical KV p99 from 26.74 seconds to 125 ms at concurrency five, and from 10.15 seconds to 288 ms at concurrency forty.

Nonblocking context-transfer polling protects the same path by reaping completed transfers even when scheduling stalls, breaking a feedback loop in which finished KV blocks stay pinned and prevent new admissions. A separate draft-cache transfer proposal was closed without merge and should not be counted as shipped TensorRT-LLM behavior.

TensorRT-LLM also moved irregular long-context work onto more efficient execution paths. Context graph producers for MiniMax-M3 capture stable sparse producers while leaving request-dependent attention eager, and per-user output throughput improved by 12.58% in its AgentX test. An open native KV-event production change reduces allocation and conversion work on the KV-aware routing path.

AgentX also exposed kernel-selection and scheduler-lifetime failures that only appear at scale and duration. Two are about which kernel gets picked. MiniMax-M3 added CuTeDSL choices to MXFP8 autotuning, widening the candidate set and improving output throughput per GPU by roughly 7 to 10% at low-concurrency aggregated points. In the opposite direction, TensorRT-LLM disabled corrupt split-K MoE tactics after they crashed five of seven AgentX runs, with no crashes in seven matched runs afterwards. A tactic that is fast and wrong is worse than one that is merely slow, and an autotuner will select it enthusiastically unless it is removed from the pool.

The other two are lifetime bugs, which are the characteristic failure of long runs rather than large ones. Sequence-slot headroom and consistent slot-indexed buffer sizing handle the transient overlap where a completing request and a newly admitted one both need a slot — a window that a steady stream of arrivals and departures hits constantly and a fixed batch never hits at all. A later attention-data-parallel dummy-request fix kept nine Qwen3.5 disaggregated cells alive where most earlier cells had failed within minutes: the difference between a configuration that benchmarks and one that survives a session.

Two open transfer changes target very long disaggregated prompts, and together they show how a fix can create the next bottleneck. In the default arrangement a decode worker cannot start until the entire prompt has been prefilled and then transferred, so two expensive phases run back to back even though the first produces its output incrementally. Pipelined KV transfer begins sending each completed prefill chunk as it lands, so transfer overlaps prefill compute and only the final chunk is on the critical path.

That change makes chunk handling frequent, which exposes work that used to happen once. Its follow-up retrieves only the block IDs belonging to the current chunk rather than the whole prompt’s block list each time. For a 128,000-token prompt split into 1,024-token chunks, that is the difference between building a 4,096-entry list once and rebuilding it 128 times for every layer group — a per-chunk cost that scales with total prompt length, which is exactly the shape that eats the gain the pipelining just bought.