AI inference
Also known as LLM inference, model serving
In plain English
You give a trained model something new, such as a prompt, image, or audio. It uses what it learned to produce an answer.
Technical definition
AI inference is the process of running a trained model on new input to produce an output. For a large language model, that usually means processing a prompt and generating tokens.
Engineering details
Training changes model weights; inference uses those weights. A production inference system wraps the model in a serving engine that schedules requests, manages memory, batches work, and runs kernels on one or more accelerators. Performance can vary with the surrounding software and hardware stack.
Why it matters
Inference performance depends on the system around the model. User experience depends on latency and interactivity, while operator economics depend on throughput, utilization, power, and hardware cost. Optimizing one dimension can make another worse.
How to read it in InferenceX
InferenceX benchmarks complete serving recipes because peak chip specifications alone cannot describe serving performance. Each curve captures a model, engine, numerical precision, parallelism strategy, GPU system, sequence length, and concurrency sweep.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM