MI300X FP8: MTP vs Off Speculative Decoding

MTP posts 370 tok/s/chip for $0.71 per million tokens at 9 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI300X FP8); Off posts 243 tok/s/chip for $1.09. MTP is 52% cheaper per token; MTP delivers 52% more tok/s/chip. Draft-token acceptance rates determine whether speculative decoding helps or hurts at a given concurrency level.

Throughput at 13 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI300X FP8): MTP hits 263 tok/s/chip, Off hits 135. Per-million costs land at $1.01 and $1.96 respectively. MTP is 95% cheaper per token; MTP delivers 95% more tok/s/chip. Speculative decoding trades extra compute on draft tokens for fewer decoding steps — the payoff depends on sequence length and batch size.

Toward the upper edge of the 5–20 tok/s/user interactivity band, at 17 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI300X FP8): MTP runs 163 tok/s/chip at $1.62/M tokens, Off runs 67 at $3.94/M. MTP is 143% cheaper per token; MTP delivers 143% more tok/s/chip. Gains from speculative decoding vary by workload; short-output prompts tend to benefit less. (Numbers reflect this URL's pinned 1k/1k · fp8 workload — changing sequence or model updates both the table and chart; the table stays pinned to this page's precision, so precision toggles in the controls affect the chart only.)