FP8
Also known as 8-bit floating point
In plain English
FP8 is a compact 8-bit way to store and calculate with model numbers, reducing memory use and often speeding up inference.
Technical definition
FP8 is a family of eight-bit floating-point formats used to reduce model storage, memory traffic, and compute cost relative to FP16 or BF16.
Engineering details
Common FP8 encodings trade exponent range against mantissa precision. Serving recipes may use FP8 for weights, activations, KV cache, or selected kernels, with scaling metadata and higher-precision accumulation where needed.
Why it matters
FP8 is broadly supported on recent NVIDIA and AMD accelerators and often serves as a stable low-precision baseline. Actual performance depends on end-to-end kernel coverage; fallback operations can erase theoretical gains.
How to read it in InferenceX
An InferenceX FP8 label covers the complete recipe. The checkpoint filename, engine, attention backend, KV-cache format, GPU generation, and MTP setting can all change the curve.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AMD MI355X GLM-5 Inference: Up to 40% Cheaper per Million Tokens than B200 on SGLang FP8
14 weeks after GLM-5 launched, AMD landed both MTP and non-MTP SGLang FP8 recipes on MI355X — fused MLA + FP8 KV cache via TileLang flips the single-node FP8 cost curve in AMD favor across most of the performance Pareto
B200 NVFP4 vs H100 FP8 on MiniMax-M2.5: Up to 8.2x Better Performance per Dollar with vLLM
vLLM PR #36307 unlocks the trtllm-gen FP8 MoE kernel for MiniMax on B200; combined with NVFP4, perf/$ scales from 4.0x at 22 tok/s/user to 8.2x at 110 on 8K/1K