Reduce-scatter
Also known as reduce scatter collective
In plain English
Reduce-scatter sums matching data from every chip and leaves each chip holding just its own slice of the combined result.
Technical definition
Reduce-scatter is a collective operation that element wise reduces tensors contributed by all devices and distributes the reduced result in shards, one shard per device.
Engineering details
When every rank computes a partial result for the same tensor, the partials must be summed. Reduce-scatter does the summation and hands each rank only the slice it will need next, avoiding the waste of giving everyone the full reduced tensor. An all-reduce is exactly a reduce-scatter followed by an all-gather, so schedulers choose between the fused and split forms depending on what the next operation actually consumes.
Why it matters
Using reduce-scatter instead of a full all-reduce halves the data each rank must receive when only a shard is needed, which matters at NVL72 scale where collective traffic competes with the model itself for interconnect bandwidth. Overlap of these collectives with compute is a defining quality of mature serving stacks.
How to read it in InferenceX
InferenceX recipes with tensor parallel sharding trigger reduction collectives in every transformer layer, and CollectiveX exists precisely to publish cross vendor measurements of these primitives at the message sizes inference actually uses.
Source material
See the concept in real benchmarks
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4