The language behind the inference curve.

You give a trained model something new, such as a prompt, image, or audio. It uses what it learned to produce an answer.

All-reduce lets every GPU solve one piece of a calculation, combines those pieces, and gives the combined result back to everyone.

All-to-all is a coordinated exchange where every GPU sends a different package of data to every other GPU.

Batching is like putting several passengers on one bus: the GPU handles multiple requests together so each trip does more useful work.

Concurrency is how many people or requests the system is serving at once.

This is the estimated infrastructure bill for producing one million tokens, the chunks of text an AI model reads and writes.

CUDA is NVIDIA’s software toolbox for making programs run on its GPUs.

Data parallelism makes multiple copies of the model and divides incoming work among them, like opening more identical checkout lanes.

Decode is the model writing its answer one token at a time after it has read the prompt.

Disaggregated inference gives prompt reading and answer writing to separate GPU teams, so each team can be tuned for its own job.

EAGLE is a particular way to draft several likely next tokens for the main model to check, which can make answers stream faster.

Expert parallelism gives different GPUs different specialist parts of a model, then sends each token to the specialists it needs.

FP4 compresses model numbers into just 4 bits. That can make inference much faster and smaller, but leaves less room for numerical detail.

FP8 is a compact 8-bit way to store and calculate with model numbers, reducing memory use and often speeding up inference.

HBM is the GPU’s small pool of extremely fast nearby memory, where model weights and working data must fit while inference runs.

The inference engine is the traffic controller behind an AI service: it keeps incoming requests moving and makes sure the GPUs do the right work at the right time.

Input length is how much the model reads; output length is how much it writes. “8K/1K” means a long prompt followed by a shorter answer.

Interactivity is how quickly one person sees new words appear after the model starts answering.

Iso-interactivity compares systems while users see words appear at the same speed. This provides an apples-to-apples view of the hardware behind the experience.

The KV cache is the model’s working memory for the current conversation. It keeps useful notes and avoids rereading everything for every new token.

Latency is how long you wait. For a streamed answer, that includes both the wait before it starts and the pauses between later words.

Memory bandwidth is the width of the pipe feeding data to the GPU’s compute units. A wider pipe keeps them from sitting idle.

A mixture-of-experts model is like a large team of specialists: it calls only the few experts best suited to each token instead of using the whole team every time.

MLA compresses the model’s notes about earlier tokens so long conversations use less memory and are cheaper to continue.

MTP lets the model guess several upcoming tokens at once and then verify them, reducing the number of slow one-token-at-a-time steps.

MXFP4 is a 4-bit format that gives small groups of numbers their own scale, helping very compact values keep enough useful range.

NVFP4 is NVIDIA’s Blackwell-optimized version of 4-bit model math, designed to move less data and use the GPU’s fastest low-precision hardware.

NVIDIA Dynamo coordinates many GPU workers. It routes requests, moves model memory, and assigns prompt reading and answer generation to the right pools.

NVLink is NVIDIA’s high-speed highway between GPUs, allowing them to cooperate much faster than over ordinary server networking.

The Pareto frontier is the line of best available tradeoffs. Each point remains viable because improving one dimension would require giving up ground on another.

Performance per dollar measures how much useful AI output the system produces for each dollar spent running it.

Prefill is the model reading and understanding your prompt before it begins writing the answer.

Prefix caching remembers the work for a repeated beginning, such as the same system prompt, so the model can skip that work next time.

Quantization stores the model’s numbers with fewer bits, making it smaller and faster to move, usually with a carefully controlled loss of precision.

ROCm is AMD’s software toolbox for running AI and other high-performance programs on AMD GPUs.

Scale-up is the ultra-fast network inside one tightly connected GPU system. Scale-out is the broader network connecting separate servers or racks.

SGLang is open-source software for serving language models quickly, with scheduling and optimization features for complex AI workloads.

Sparse attention lets the model look back at only the most useful parts of a long context instead of rereading every earlier token.

Speculative decoding lets a cheaper helper draft several tokens ahead, then asks the full model to approve them together instead of generating each one separately.

Tensor parallelism splits one large calculation across several GPUs so they solve it together.

TensorRT-LLM is NVIDIA’s optimized software stack for getting high inference performance from NVIDIA GPUs.

Throughput is how much total work the system gets done each second across everyone using it.

TPOT is the gap between each new piece of a streamed answer. Smaller gaps make the response feel faster and smoother.

TTFT is the “thinking…” pause between sending your prompt and seeing the first piece of the answer.

Tokens per megawatt asks how much AI output a data center can produce from a fixed amount of available power.

TCO covers the hardware purchase plus the cost of powering, cooling, networking, and operating it over time.

vLLM is open-source software that organizes requests and GPU memory so language models can serve many users efficiently.

Wide expert parallelism spreads a model’s specialists across many GPUs, giving each GPU less expert data to hold and move.