FP4
Also known as 4-bit floating point
In plain English
FP4 compresses model numbers into just 4 bits. That can make inference much faster and smaller, but leaves less room for numerical detail.
Technical definition
FP4 refers to four-bit floating-point formats used for very low-precision model representation and accelerated matrix operations.
Engineering details
Four-bit formats roughly halve weight storage and traffic again relative to FP8, but their tiny value space requires carefully chosen scaling and hardware-specific kernels. The FP4 label covers several concrete formats.
Why it matters
For memory-bound LLM inference, reducing weight bytes can deliver large throughput and capacity gains. Model quality and unsupported operations must be checked because aggressive precision reduction can also introduce error or fallback overhead.
How to read it in InferenceX
InferenceX identifies concrete recipe formats such as NVFP4 and MXFP4 where possible and validates representative configurations. Each FP4 line still has its own numerical and operational behavior.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
B200 NVFP4 vs H200 INT4 on Kimi K2.5/K2.6: Up to 2.95x Better Performance per Dollar
On vLLM 8K/1K the NVFP4 path on B200 is 2.71x–2.95x cheaper per million tokens than H200 INT4 across the entire 30–90 tok/s/user serving band, and 2.45x–2.74x cheaper than B200 INT4 on the same silicon. Both factors decompose cleanly into B200's HBM bandwidth, HBM capacity, and NVFP4 tensor cores
MI355X DeepSeek-V4-Pro on SGLang: 110.5x Throughput per GPU in 26 Days
The amd/deepseek_v4 side branch shipped TileLang attention indexer, Triton sparse MLA, fused RoPE/Hadamard, FlyDSL MoE, and FP4 weights across 31 performance optimizations PRs — lifting first-light 20 tok/s/GPU at 2.4 tok/s/user into 2,256 tok/s/GPU at 9.4 tok/s/user on 8K/1K, with both throughput and interactivity climbing together
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool