Memory bound vs compute bound
Also known as bandwidth bound, compute limited
In plain English
A workload is compute bound when the math units are the bottleneck and memory bound when waiting on data movement is the bottleneck.
Technical definition
A kernel is compute bound when its runtime is set by arithmetic throughput and memory bound when its runtime is set by how fast operands move between memory and the compute units.
Engineering details
Every kernel has an arithmetic intensity, the ratio of operations to bytes touched. If that ratio is below the hardware balance point, the memory system saturates before the math units do. LLM prefill runs large matrix multiplies with high intensity and tends to be compute bound, while decode reads the entire working set of weights and KV cache to produce one token per request and is usually memory bound.
Why it matters
The binding resource decides which hardware spec matters. Memory bound decode explains why HBM capacity and bandwidth headline every accelerator launch, why quantization speeds up decode by shrinking bytes moved, and why a chip with modest FLOPs but fast memory can win interactive serving.
How to read it in InferenceX
InferenceX sweeps concurrency, which walks a system between regimes: low concurrency decode is bandwidth limited while high concurrency batching pushes toward compute limits. The shape of each throughput versus interactivity curve reflects where that transition happens for a given recipe.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine