INT4
Also known as 4-bit integer, W4A16
In plain English
INT4 stores weights in four-bit integers, shrinking the model enough to move far less memory per token on hardware without native 4-bit floats.
Technical definition
INT4 is a four-bit integer format used mainly for weight quantization, typically with a higher-precision scale per group of values.
Engineering details
Integer formats spread their values evenly, unlike floating point, so INT4 depends on grouped scaling factors to track the local range of each block of weights. Activations commonly stay at higher precision, and the matrix multiply dequantizes on the fly, which makes the technique a memory movement optimization more than an arithmetic one.
Why it matters
It matters most where 4-bit floating point has no hardware support. On such parts INT4 is the practical route to 4-bit weights, though it usually needs more calibration care than a native format and its accuracy has to be checked rather than assumed.
How to read it in InferenceX
InferenceX treats INT4 as its own precision key alongside FP4, FP8, and BF16, and precision is part of the recipe rather than a display option. Compare INT4 against a native FP4 recipe only with the accuracy evaluations in view, since the formats are not interchangeable.
Source material
See the concept in real benchmarks
B200 NVFP4 vs H200 INT4 on Kimi K2.5/K2.6: Up to 2.95x Better Performance per Dollar
On vLLM 8K/1K the NVFP4 path on B200 is 2.71x–2.95x cheaper per million tokens than H200 INT4 across the entire 30–90 tok/s/user serving band, and 2.45x–2.74x cheaper than B200 INT4 on the same silicon. Both factors decompose cleanly into B200's HBM bandwidth, HBM capacity, and NVFP4 tensor cores
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18