Mixture of experts
Also known as sparse MoE
In plain English
A mixture-of-experts model is like a large team of specialists: it calls only the few experts best suited to each token instead of using the whole team every time.
Technical definition
A mixture-of-experts model contains many feed-forward expert networks but routes each token through only a selected subset.
Engineering details
A router scores experts for each token, and top-k routing activates the chosen experts plus any shared experts. This lets total parameter count grow much larger than the computation used for one token.
Why it matters
MoE inference trades arithmetic sparsity for systems complexity. Expert weights still consume memory, routing can become imbalanced, and distributed deployments require all-to-all communication for dispatch and combine.
How to read it in InferenceX
InferenceX covers models with hundreds of experts and reports both total and activated parameters where relevant. TP, EP, DP, precision, and network topology determine whether MoE sparsity becomes a real serving advantage.
Source material
See the concept in real benchmarks
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.
B200 NVFP4 vs H200 INT4 on Kimi K2.5/K2.6: Up to 2.95x Better Performance per Dollar
On vLLM 8K/1K the NVFP4 path on B200 is 2.71x–2.95x cheaper per million tokens than H200 INT4 across the entire 30–90 tok/s/user serving band, and 2.45x–2.74x cheaper than B200 INT4 on the same silicon. Both factors decompose cleanly into B200's HBM bandwidth, HBM capacity, and NVFP4 tensor cores
B200 NVFP4 vs H100 FP8 on MiniMax-M2.5: Up to 8.2x Better Performance per Dollar with vLLM
vLLM PR #36307 unlocks the trtllm-gen FP8 MoE kernel for MiniMax on B200; combined with NVFP4, perf/$ scales from 4.0x at 22 tok/s/user to 8.2x at 110 on 8K/1K
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18