Remote direct memory access
Also known as RDMA, RoCE, GPUDirect RDMA
In plain English
RDMA lets one machine read or write the memory of another directly over the network, without either CPU copying data along the way.
Technical definition
Remote direct memory access is a networking capability where the network adapter moves data straight between the memories of two machines, bypassing operating system and CPU copy overhead.
Engineering details
A conventional network stack copies data through kernel buffers on both ends, burning CPU cycles and latency. RDMA adapters transfer directly between registered memory regions, and GPUDirect extends this so adapters write straight into accelerator HBM. InfiniBand has RDMA built in, while RoCE carries the same verbs over Ethernet. Collective libraries such as NCCL and RCCL, and KV cache transfer paths in disaggregated serving, are built on these primitives.
Why it matters
RDMA is the floor the whole distributed AI stack stands on: without it, cross node collectives and cache transfers would bottleneck on CPUs long before saturating the links. The RoCE variant is what allows Ethernet based clusters to compete with InfiniBand at lower cost.
How to read it in InferenceX
Every InferenceX multi node result depends on RDMA transports underneath its collectives and, for disaggregated recipes, underneath prefill to decode KV movement, so transport maturity is part of what separates otherwise similar cluster results.
Source material
See the concept in real benchmarks
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2
DeepSeek V4 Pro on AgentX: GB200 vs GB300 Rack-Scale Disaggregation
Both lean on PD disagg, GB300 adds DEP32 wide-EP decode, and the gap shows up in first-token latency rather than token rate