Dense model
Also known as dense transformer, dense LLM
In plain English
A dense model applies every one of its parameters to every token it processes, unlike sparse models that route tokens to a few experts.
Technical definition
A dense model is a neural network in which all weights participate in every forward pass, so compute per token scales directly with total parameter count.
Engineering details
Dense transformers are the simpler design: every layer processes every token with all of its weights. That makes their behavior predictable, their parallelism straightforward, and their quality per parameter strong, but serving cost grows linearly with size. Mixture of experts models break that link by activating a fraction of their weights per token, which is why the largest frontier models are sparse while small and mid size models often stay dense.
Why it matters
The dense versus sparse choice drives serving strategy. Dense models fit on fewer chips and avoid expert routing complexity, while sparse models buy more quality per unit of compute at the price of much larger memory footprints and heavier cross chip communication.
How to read it in InferenceX
InferenceX coverage centers on the large sparse models operators actually deploy at the frontier, and dense baselines such as Llama class models provide contrast in how tensor parallelism and memory pressure behave without expert routing.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM