Arithmetic intensity
Also known as operational intensity, compute to memory ratio
In plain English
Arithmetic intensity is how much math a computation does per byte it moves, which decides whether the chip or its memory is the limit.
Technical definition
Arithmetic intensity is the ratio of arithmetic operations performed to bytes moved between memory and the compute units.
Engineering details
Prefill processes many tokens against one set of weights, so it reuses each loaded byte heavily and is usually compute bound. Decode emits one token per step per sequence and must still read the weights and the KV cache, so it moves a great deal of data for very little arithmetic and is usually memory bandwidth bound.
Why it matters
The two phases are limited by different parts of the chip, which is why a specification sheet with impressive peak FLOPS can disappoint on decode and why batching helps: grouping sequences raises intensity by reusing each weight read across more tokens.
How to read it in InferenceX
The split explains recurring shapes in the data. Low interactivity points run large batches at high intensity and approach compute limits, while the high interactivity end runs small batches and tracks memory bandwidth, so bandwidth-rich parts often win there despite lower peak throughput.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine