All-reduce
In plain English
All-reduce lets every GPU solve one piece of a calculation, combines those pieces, and gives the combined result back to everyone.
Technical definition
All-reduce is a collective communication operation that combines values from every participating rank and returns the reduced result to every rank.
Engineering details
Tensor-parallel layers use all-reduce to assemble partial matrix-operation results. The collective may sum or otherwise reduce values while moving data through an optimized ring, tree, or fabric-specific algorithm.
Why it matters
Because TP can require collectives at many layers for every generated token, all-reduce latency and bandwidth set a hard scaling limit. Small decode batches are especially sensitive to fixed communication latency.
How to read it in InferenceX
A higher TP width can add compute and memory bandwidth but also expands the collective group. Results must show whether the interconnect turns that larger group into a net gain.