BF16
Also known as bfloat16, brain float 16
In plain English
BF16 is the 16-bit format most models are trained in, and serving in it is the accuracy reference the quantized recipes are measured against.
Technical definition
BF16 is a 16-bit floating point format with the same exponent range as FP32 and a reduced mantissa, widely used for training and as an unquantized serving baseline.
Engineering details
Keeping the FP32 exponent range makes BF16 tolerant of the value distributions that appear in transformer activations, so conversion rarely needs the scaling machinery narrower formats require. The tradeoff is precision rather than range, and the format is twice the size of FP8 and four times that of a 4-bit format.
Why it matters
Its role in a benchmark is usually as a reference point. Weight reads dominate decode, so a BF16 recipe moves far more memory per token than a quantized one and tends to sit lower on the throughput curve while defining the accuracy the others are compared against.
How to read it in InferenceX
InferenceX carries BF16 as a precision key and reports peak BF16 dense throughput per accelerator in the specs pages. Quantized recipes are validated with accuracy evaluations rather than assumed lossless, which is what makes a BF16 comparison meaningful.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time — Huawei, GB300 NVL72, MI355X, B200
Day 0 Inference Performance, InferenceX, 100x performance improvement in 26 Days, Cost per Million Tokens, Huawei 950DT Inference Trace Analysis