Tensor parallelism
In plain English
Tensor parallelism splits one large calculation across several GPUs so they solve it together.
Technical definition
Tensor parallelism shards individual tensor operations and model weight matrices across multiple accelerators.
Engineering details
Each layer executes cooperatively across ranks. Partial results must be combined with collective communication, commonly all-reduce operations after parallel matrix multiplications.
Why it matters
TP lets a model fit across devices and can improve low-batch interactivity by pooling compute and memory bandwidth. Communication occurs frequently, so scaling eventually runs into the bandwidth and latency of the interconnect.
How to read it in InferenceX
InferenceX recipe labels such as TP=4 or TP=8 state how many ranks participate in each tensor-parallel group. Compare TP together with EP, DP, node count, and network domain.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
AMD MI355X Qwen3.5 397B-A17B Inference: Up to 19x Throughput per GPU in 3 Months on SGLang FP8
From v0.5.8 (Feb) → v0.5.10rc0 (Apr) → v0.5.12 (May), three AITER kernel landings on MI355X plus a TP=8 → TP=2/TP=4 retune push Qwen3.5 8k/1k peak from 1.3k to 6.4k tok/s/GPU and extend the curve out to 75 tok/s/user