Weight-only quantization
Also known as W4A16, AWQ, GPTQ
In plain English
Weight-only quantization compresses just the stored model weights to low precision while the math still runs in higher precision formats.
Technical definition
Weight-only quantization stores model weights in a low bit format such as 4 bit integers while keeping activations and arithmetic at higher precision, as in the W4A16 pattern.
Engineering details
Weights are static and can be carefully quantized offline with methods such as GPTQ and AWQ, which choose scales and orderings that minimize quality loss. Activations are dynamic and harder to compress, so leaving them at 16 bits sidesteps their outlier problem. At serve time the kernel dequantizes weights on the fly, so memory traffic shrinks even though the multiply accumulate math itself does not get faster.
Why it matters
Because decode is memory bound, cutting weight bytes directly speeds up token generation and lets larger models fit on fewer chips. Weight-only methods made large open models runnable on modest hardware and remain the standard recipe when activation quantization would cost too much quality.
How to read it in InferenceX
InferenceX distinguishes weight-only configurations from full low precision paths in its precision labels, since a W4A16 recipe and an NVFP4 recipe make very different claims about which hardware units and bandwidth budgets produced a curve.
Source material
See the concept in real benchmarks
B200 NVFP4 vs H200 INT4 on Kimi K2.5/K2.6: Up to 2.95x Better Performance per Dollar
On vLLM 8K/1K the NVFP4 path on B200 is 2.71x–2.95x cheaper per million tokens than H200 INT4 across the entire 30–90 tok/s/user serving band, and 2.45x–2.74x cheaper than B200 INT4 on the same silicon. Both factors decompose cleanly into B200's HBM bandwidth, HBM capacity, and NVFP4 tensor cores
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18