All-to-all
In plain English
All-to-all is a coordinated exchange where every GPU sends a different package of data to every other GPU.
Technical definition
All-to-all is a collective pattern in which every participating rank sends distinct data to every other rank.
Engineering details
Expert-parallel MoE layers use an all-to-all dispatch to send tokens to their selected experts and another combine operation to return expert outputs. Traffic volume and imbalance depend on token routing.
Why it matters
All-to-all is more demanding than simple point-to-point transfers and can become network bound as EP grows. Specialized kernels overlap communication with compute and optimize token packing to keep the fabric busy.
How to read it in InferenceX
Rack-scale NVLink can keep wide-EP all-to-all traffic inside the scale-up domain. Multi-node recipes over InfiniBand or RoCE must overcome a much lower per-GPU scale-out bandwidth.
Source material
See the concept in real benchmarks
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM