Scale-up vs. scale-out networking
Also known as scale-up domain, scale-out fabric
In plain English
Scale-up is the ultra-fast network inside one tightly connected GPU system. Scale-out is the broader network connecting separate servers or racks.
Technical definition
Scale-up networking connects accelerators inside one tightly coupled system, while scale-out networking connects multiple systems or racks into a larger cluster.
Engineering details
Scale-up fabrics such as NVLink offer very high per-GPU bandwidth and low latency for fine-grained collectives. Scale-out fabrics such as InfiniBand or RoCE reach more machines but usually provide much less bandwidth per accelerator.
Why it matters
Distributed inference crosses both domains. Frequent TP or EP collectives benefit disproportionately from staying inside scale-up, while coarser request routing and some prefill/decode transfers can tolerate scale-out.
How to read it in InferenceX
System topology determines the communication domain. A B200 in an eight-GPU node and a GB200 NVL72 expose related silicon through different scale-up group sizes.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4