Data parallelism
In plain English
Data parallelism makes multiple copies of the model and divides incoming work among them, like opening more identical checkout lanes.
Technical definition
Data parallelism runs replicated model or layer groups on multiple ranks and distributes requests or tokens among those replicas.
Engineering details
Classic DP duplicates the complete model. In LLM serving, hybrid forms such as data-parallel attention can replicate attention while expert weights use a different sharding strategy. Each replica handles separate work with less per-layer synchronization than TP.
Why it matters
DP scales aggregate capacity cleanly when weights fit, but replication consumes memory and repeats weight reads. Load balancing and cache locality determine how evenly the replicas are used.
How to read it in InferenceX
Modern MoE deployments combine DP, TP, and EP. Read the DP count together with the other two dimensions.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
MI355X DeepSeek-V4-Pro on SGLang: 110.5x Throughput per GPU in 26 Days
The amd/deepseek_v4 side branch shipped TileLang attention indexer, Triton sparse MLA, fused RoPE/Hadamard, FlyDSL MoE, and FP4 weights across 31 performance optimizations PRs — lifting first-light 20 tok/s/GPU at 2.4 tok/s/user into 2,256 tok/s/GPU at 9.4 tok/s/user on 8K/1K, with both throughput and interactivity climbing together
GB200 NVL72 vs B200 on Kimi K2.5: 3.1x from Wide EP vLLM
Rack scale NVLink on NVL72 lets Dynamo vLLM run Kimi K2.5 wide EP up to Decode EP 16, taking peak throughput from 4,021 to 12,587 tok/s/GPU on 8k/1k NVFP4