MI355X: FP4 vs FP8 Precision Comparison

FP4 posts 2160 tok/s/GPU for $0.19 per million tokens at 55 tok/s/user on Llama 3.3 70B (MI355X); FP8 posts 2185 tok/s/GPU for $0.18. Cost per token is essentially tied; FP8 delivers 1% more tok/s/GPU. Quantization-level accuracy differences are tracked on the evaluation tab.

Throughput at 76 tok/s/user on Llama 3.3 70B (MI355X): FP4 hits 1069 tok/s/GPU, FP8 hits 1504. Per-million costs land at $0.38 and $0.27 respectively. FP8 is 44% cheaper per token; FP8 delivers 41% more tok/s/GPU. The cost-throughput tradeoff from lower precision is only part of the picture — see the evaluation page for accuracy data.

Toward the upper edge of the 36–116 tok/s/user interactivity band, at 96 tok/s/user on Llama 3.3 70B (MI355X): FP4 runs 525 tok/s/GPU at $0.74/M tokens, FP8 runs 750 at $0.55/M. FP8 is 36% cheaper per token; FP8 delivers 43% more tok/s/GPU. Precision changes affect both inference speed and model quality — consult the evaluation tab for accuracy benchmarks. (Numbers reflect the default 1k/1k selection for this URL — table and chart below update if you change sequence or model in the controls. Each side uses the best available serving configuration for that precision, which may include speculative decoding such as MTP where recipes exist — the same convention as the other comparison pages.)