All-gather
Also known as allgather, gather collective
In plain English
All-gather is a group communication step where every chip ends up holding the combined data that started out split across all of them.
Technical definition
All-gather is a collective operation in which each participating device contributes its shard and every device receives the concatenation of all shards.
Engineering details
Sharded execution constantly needs to reassemble full tensors: weight shards before a matrix multiply in some tensor parallel layouts, or per device activations before an operation that needs the whole hidden state. All-gather moves each shard to every rank, typically over a ring or tree schedule, and its cost grows with tensor size and the number of participants. It is the inverse companion of reduce-scatter, and the two composed together form an all-reduce.
Why it matters
Together with all-reduce and all-to-all, all-gather is one of the handful of collectives that decide whether a parallelism plan scales. Its latency sits on the critical path of every layer that uses it, which is why scale up bandwidth between chips is so heavily marketed.
How to read it in InferenceX
Every multi chip recipe InferenceX benchmarks exercises collectives through its parallelism plan, and the CollectiveX workstream measures operations like this directly across vendors so communication behavior can be compared outside full model runs.
Source material
See the concept in real benchmarks
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM