GPU utilization
Also known as chip utilization, accelerator utilization
In plain English
GPU utilization measures how busy an accelerator is, though the common percentage from monitoring tools can look high while real efficiency stays low.
Technical definition
GPU utilization is the share of time or capability an accelerator spends doing useful work, reported anywhere from coarse busy percentages to strict measures like model FLOPs utilization.
Engineering details
The utilization number in basic monitoring tools only says a kernel was resident, not that it used the chip well. A kernel occupying one compute unit still reads as busy. Stricter measures compare delivered arithmetic or bandwidth against hardware peaks. In serving, utilization is also shaped by traffic: idle gaps between requests, low concurrency, and tool call pauses in agent sessions all leave paid-for silicon idle.
Why it matters
Fleet economics hinge on utilization. The gap between a well batched, well scheduled deployment and a naive one is often several times the cost per token on identical hardware, which is why serving software and request routing get as much attention as the chips themselves.
How to read it in InferenceX
InferenceX benchmarks report delivered throughput per chip at each interactivity level rather than a utilization percentage, so utilization differences between engines, precisions, and parallelism plans show up directly as separation between curves on the same hardware.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure