Model FLOPs utilization
Also known as MFU, model bandwidth utilization, MBU
In plain English
MFU compares the useful math a model actually performed against the maximum the chip could theoretically perform, giving an efficiency percentage.
Technical definition
Model FLOPs utilization is the ratio of the floating point operations a model logically requires to the peak operations the hardware could deliver in the same wall clock time.
Engineering details
Peak TFLOP/s figures assume every tensor core is busy every cycle, which never happens in serving. Kernel launch gaps, memory stalls, communication waits, and imperfect batching all leave compute idle. MFU folds all of that into one number. Decode is usually memory bound, so decode MFU is naturally low, and the bandwidth analog MBU is often the more honest efficiency measure for token generation.
Why it matters
MFU separates hardware capability from software maturity. A chip with enormous peak FLOPs but weak kernels can lose to a slower chip that keeps its units fed. Rising MFU on fixed hardware is the signature of software progress, which is where most inference performance gains come from.
How to read it in InferenceX
InferenceX tracks delivered tokens per chip over time rather than reporting MFU directly, and the repeated pattern of large gains on unchanged hardware, such as order of magnitude improvements within weeks of a model release, is utilization being recovered by better software.
Source material
See the concept in real benchmarks
MI355X DeepSeek-V4-Pro on SGLang: 110.5x Throughput per GPU in 26 Days
The amd/deepseek_v4 side branch shipped TileLang attention indexer, Triton sparse MLA, fused RoPE/Hadamard, FlyDSL MoE, and FP4 weights across 31 performance optimizations PRs — lifting first-light 20 tok/s/GPU at 2.4 tok/s/user into 2,256 tok/s/GPU at 9.4 tok/s/user on 8K/1K, with both throughput and interactivity climbing together
DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time — Huawei, GB300 NVL72, MI355X, B200
Day 0 Inference Performance, InferenceX, 100x performance improvement in 26 Days, Cost per Million Tokens, Huawei 950DT Inference Trace Analysis
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM