AgentX industry impact · Transfer engine
Mooncake
Mooncake serves Moonshot’s Kimi production traffic along with production traffic at many labs, and is a transfer engine underneath disaggregated vLLM and SGLang configurations. Until recently its AMD support stopped short of both RDMA registration and installable packages.
- GPU-direct RDMA path on ROCm
- HIP dmabuf
- covered by one architecture-agnostic wheel
- gfx942 + gfx950
- tag-triggered publish matrix
- Python 3.10–3.13
GPU-direct RDMA registration on ROCm
Registering GPU memory for RDMA on NVIDIA uses either the nvidia-peermem kernel module or an exported dmabuf file descriptor. AMD has no nvidia-peermem equivalent, so GPU-direct RDMA had no path at all and deployments fell back to staging KV through host DRAM.
A HIP dmabuf registration branch adds the mirror of the existing CUDA dmabuf path, exporting through ROCm instead of the CUDA handle call, and resolving the true allocation base first because caching allocators pack tensors at an offset inside a larger allocation. Host memory still registers directly.
Upstream pull requests
Support that cannot be installed is not support
Mooncake published CUDA and MUSA wheels but no ROCm package, so AMD users built the engine from source inside every image. A ROCm wheel, CI, and release path publishes mooncake-transfer-engine-rocm to PyPI alongside them. This workstream came from Andy Luo, an AMD engineer, who noticed the pattern while dogfooding agentic workloads with AgentX: building Mooncake from source is not a first-class ROCm experience.
The transfer engine has no device kernels and does not depend on torch, so one architecture-agnostic wheel covers gfx942 and gfx950, and the ROCm runtime is bound at load time rather than vendored — which means the same wheel works unmodified in both the upstream vLLM ROCm image and the SGLang ROCm image. That was verified as a full cross product: MI300X and MI355X, each under vllm/vllm-openai-rocm and lmsysorg/sglang, running the master binary and a HIP buffer transfer test with data verification. The pull request adds a tag-triggered publish across Python 3.10 through 3.13.
An open follow-up adds a self-hosted two-node MI350X external prefill and decode tier so the ROCm disaggregated path is exercised on real hardware rather than only compiled. Together these mean an AMD AgentX run can now install the transfer engine and the KV-cache layer from published artifacts, into stock upstream images, and move KV directly between GPU memory and the fabric.
Upstream pull requests
References
Other projects
Inference engine
vLLM
Hybrid-attention prefix retention, CPU KV offload for hybrid models, and a narrowed store and load path.
Read the optimizations →Inference engine
SGLang
Sliding-window allocation, HiCache hybrid offload, runtime-scalar context length, and cache-aware DP routing.
Read the optimizations →Inference engine
TensorRT-LLM
Boundary-aware incremental tokenization, disaggregated KV descriptor coalescing, and scheduler-lifetime fixes.
Read the optimizations →Inference engine
AMD ATOM
Sparse checkpoint retention, recurrent-state checkpoints, CPU offload ownership, and long-prefill parallelism.
Read the optimizations →Kernels
ROCm AITER
Context-parallel process groups, 64-bit addressing for large cache pools, and persistent MLA decode kernels.
Read the optimizations →Router and orchestration
NVIDIA Dynamo
Batched KV matching, request-lease ownership, cheaper router state, and a leaner request plane.
Read the optimizations →KV-cache layer
LMCache
Chunked external-cache loading, hybrid-group storage, AMD Instinct enablement, and DCP-aware offload.
Read the optimizations →