TPU7x FP8: MTP vs Off Speculative Decoding
Speculative decoding comparison of MTP versus Off on TPU7x FP8 (Google Ironwood) running Qwen 3.5 397B-A17B. Throughput, cost, and interactivity differences across LLM workloads. Use the chart controls below to switch sequences and metrics — same interactions as the main inference chart.
MTP acceptance-rate implementations differ across inference engines. Points from different engines are not directly comparable on the same curve — throughput and cost at matched interactivity may reflect engine-level differences rather than pure speculative decoding gains. Interpret cross-engine comparisons with caution.
Off: 5664 tok/s/chip, $0.05 per million tokens at 59 tok/s/user on Qwen 3.5 397B-A17B (TPU7x FP8). MTP is unmeasured here.
At 111 tok/s/user on Qwen 3.5 397B-A17B (TPU7x FP8), Off delivers 3559 tok/s/chip at $0.08 per million tokens; MTP hasn't been benchmarked at this target.
Off hits 2468 tok/s/chip for $0.12 per million tokens at 163 tok/s/user on Qwen 3.5 397B-A17B (TPU7x FP8). No MTP data at this operating point. (Numbers reflect this URL's pinned 8k/1k · fp8 workload — changing sequence or model updates both the table and chart; the table stays pinned to this page's precision, so precision toggles in the controls affect the chart only.)
| Metric | Interactivity (tok/s/user) | Interactivity (tok/s/user) | Interactivity (tok/s/user) |
|---|---|---|---|
| Throughput (tok/s/chip) | MTP:—Off:5663.6 | MTP:—Off:3559.1 | MTP:—Off:2467.8 |
| Cost ($/M tok) | MTP:—Off:$0.051 | MTP:—Off:$0.080 | MTP:—Off:$0.116 |
| tok/s/MW | MTP:—Off:4692296 | MTP:—Off:2948752 | MTP:—Off:2044583 |
| Concurrency | MTP:—Off:~45 | MTP:—Off:~15 | MTP:—Off:~7 |
Inference Performance
Agentic inference metrics from the AgentX scenario and fixed-sequence inference metrics across models, hardware configurations, and serving parameters.