The language behind the inference curve.

Acceptance length is how many drafted tokens the full model actually approves per verification step, which is what decides whether speculation pays off.

Active parameters are the weights a mixture of experts model actually uses for each token, a small slice of its much larger total size.

This is the request pattern created when a coding agent reads a repository, edits code, runs tools, and revisits the model until the task is done.

Agentic inference serves an AI system that works through a task over many model requests, often using tools and delegating work along the way.

AgentX is the InferenceX workload for testing how inference systems serve complete long-context, multi-turn coding-agent sessions.

You give a trained model something new, such as a prompt, image, or audio. It uses what it learned to produce an answer.

AIPerf is the vendor-neutral client that sends the benchmark traffic, reconstructing recorded agent sessions and timing every request.

AITER is AMD’s tuned kernel library, the layer that decides how fast attention and matrix work actually run on Instinct chips.

All-gather is a group communication step where every chip ends up holding the combined data that started out split across all of them.

All-reduce lets every chip solve one piece of a calculation, combines those pieces, and gives the combined result back to everyone.

All-to-all is a coordinated exchange where every chip sends a different package of data to every other chip.

Arithmetic intensity is how much math a computation does per byte it moves, which decides whether the chip or its memory is the limit.

ATOM is AMD’s own inference engine, its answer to a vendor runtime rather than an upstream open-source one.

Batching is like putting several passengers on one bus: the chip handles multiple requests together so each trip does more useful work.

BF16 is the 16-bit format most models are trained in, and serving in it is the accuracy reference the quantized recipes are measured against.

Block scaling gives each small group of low precision numbers its own shared scale factor, recovering range that tiny formats lack on their own.

Chunked prefill reads a long prompt in slices instead of all at once, so other users keep receiving tokens while it is being read.

In a closed-loop benchmark, each simulated user waits for one step to finish before sending the next step in that session.

Concurrency is how many people or requests the system is serving at once.

Context parallelism splits one long prompt across several chips so they share the work of reading it and of scanning its stored attention state.

The context window is the maximum number of tokens a model can consider at once, covering both the input and everything generated so far.

Continuous batching lets new requests join a running batch the moment old ones finish, instead of waiting for the whole batch to complete.

This is the estimated infrastructure bill for producing one million tokens, the chunks of text an AI model reads and writes.

CUDA is NVIDIA’s software toolbox for making programs run on its chips.

CUDA graphs record a whole sequence of chip operations once and replay it as a unit, removing the per-step cost of launching each one.

CUTLASS is an NVIDIA template library that provides the building blocks for writing matrix multiply kernels that approach peak hardware speed.

Data parallelism makes multiple copies of the model and divides incoming work among them, like opening more identical checkout lanes.

DP attention gives each rank its own slice of the attention work and its own cache, instead of every rank holding a copy of the same thing.

Decode is the model writing its answer one token at a time after it has read the prompt.

A dense model applies every one of its parameters to every token it processes, unlike sparse models that route tokens to a few experts.

Disaggregated inference gives prompt reading and answer writing to separate chip teams, so each team can be tuned for its own job.

A draft model is the small fast model in speculative decoding that guesses several upcoming tokens for the large model to verify in one pass.

This metric asks how fast a whole answer arrives, counting the wait before the first word as well as the streaming speed after it.

EAGLE is a particular way to draft several likely next tokens for the main model to check, which can make answers stream faster.

Energy per token is how much electricity the system spends to produce one token, the power-side counterpart of cost per token.

Expert parallelism gives different chips different specialist parts of a model, then sends each token to the specialists it needs.

FlashAttention computes exact attention in fast on-chip memory tiles, avoiding the huge intermediate matrix that made attention slow and memory hungry.

FlashInfer is a library of attention kernels that serving engines call instead of writing their own attention implementations.

FP4 compresses model numbers into just 4 bits. That can make inference much faster and smaller, but leaves less room for numerical detail.

FP8 is a compact 8-bit way to store and calculate with model numbers, reducing memory use and often speeding up inference.

GEMM is the general matrix multiply operation, the single computation that consumes most of the arithmetic in training and serving neural networks.

Goodput counts only the work that meets your latency target, so a fast-looking system that misses deadlines gets no credit for it.

GPU utilization measures how busy an accelerator is, though the common percentage from monitoring tools can look high while real efficiency stays low.

Grouped-query attention lets several query heads share one set of key and value heads, shrinking the KV cache without giving up much quality.

HBM is the chip’s small pool of extremely fast nearby memory, where model weights and working data must fit while inference runs.

A hybrid model mixes attention types across its layers, so its cache is several different kinds of state rather than one uniform block.

The inference engine is the traffic controller behind an AI service: it keeps incoming requests moving and makes sure the chips do the right work at the right time.

InfiniBand is a high speed, low latency network fabric that connects servers in AI clusters, carrying traffic between nodes that NVLink cannot reach.

Input length is how much the model reads; output length is how much it writes. “8K/1K” means a long prompt followed by a shorter answer.

INT4 stores weights in four-bit integers, shrinking the model enough to move far less memory per token on hardware without native 4-bit floats.

INT8 stores numbers as 8 bit integers with a scale factor, halving memory versus 16 bit formats and doubling math rates on supporting hardware.

Interactivity is how quickly one person sees new words appear after the model starts answering.

Iso-interactivity compares systems while users see words appear at the same speed. This provides an apples-to-apples view of the hardware behind the experience.

Kernel fusion merges several small chip operations into one, so intermediate data stays in fast memory instead of bouncing through HBM.

The KV cache is the model’s working memory for the current conversation. It keeps useful notes and avoids rereading everything for every new token.

A KV cache manager is the component that stores attention state outside the chips and decides what to keep, evict, and fetch back.

KV cache offload parks attention state the chips cannot hold in host memory, so a long session can be resumed instead of recomputed.

This stores the conversation cache in a smaller format, so a chip holds more context and reads it back faster during generation.

KV-aware routing sends a request to the worker that already holds its conversation state, instead of to whichever worker is least busy.

Latency is how long you wait. For a streamed answer, that includes both the wait before it starts and the pauses between later words.

Linear attention keeps a fixed-size running summary instead of every past token, so its memory does not grow as the conversation does.

Memory bandwidth is the width of the pipe feeding data to the chip’s compute units. A wider pipe keeps them from sitting idle.

A workload is compute bound when the math units are the bottleneck and memory bound when waiting on data movement is the bottleneck.

A mixture-of-experts model is like a large team of specialists: it calls only the few experts best suited to each token instead of using the whole team every time.

MFU compares the useful math a model actually performed against the maximum the chip could theoretically perform, giving an efficiency percentage.

MLA compresses the model’s notes about earlier tokens so long conversations use less memory and are cheaper to continue.

MTP lets the model guess several upcoming tokens at once and then verify them, reducing the number of slow one-token-at-a-time steps.

MXFP4 is a 4-bit format that gives small groups of numbers their own scale, helping very compact values keep enough useful range.

NCCL is the NVIDIA library that moves data between chips during collective operations, with RCCL as its AMD counterpart.

NVFP4 is NVIDIA’s Blackwell-optimized version of 4-bit model math, designed to move less data and use the chip’s fastest low-precision hardware.

NVIDIA Dynamo coordinates many chip workers. It routes requests, moves model memory, and assigns prompt reading and answer generation to the right pools.

NVL72 is a rack where 72 accelerators share one high-speed fabric, so they behave more like a single large machine than a cluster.

NVLink is NVIDIA’s high-speed highway between chips, allowing them to cooperate much faster than over ordinary server networking.

Offline inference processes a large pile of requests with no user waiting, so the only goal is maximum tokens per dollar, not latency.

PagedAttention stores the KV cache in small fixed size blocks, like virtual memory pages, so cache memory is not wasted on unused space.

The Pareto frontier is the line of best available tradeoffs. Each point remains viable because improving one dimension would require giving up ground on another.

Performance per dollar measures how much useful AI output the system produces for each dollar spent running it.

Pipeline parallelism gives each chip a different slice of the layers, passing activations along the chain instead of splitting each layer.

PUE measures how much total datacenter power is consumed for every watt that actually reaches the computing equipment inside it.

Prefill is the model reading and understanding your prompt before it begins writing the answer.

The hit rate is the share of prompt tokens served from cache instead of being recomputed, which on long sessions is most of the prompt.

Prefix caching remembers the work for a repeated beginning, such as the same system prompt, so the model can skip that work next time.

Quantization stores the model’s numbers with fewer bits, making it smaller and faster to move, usually with a carefully controlled loss of precision.

RadixAttention keeps completed KV cache in a radix tree keyed by token content, so any new request can reuse the longest matching prefix.

A reasoning model generates long hidden chains of thought before answering, trading extra output tokens for better results on hard problems.

A recipe is the complete set of choices behind one curve: which model, engine, image, precision, parallelism, and workload were run.

Reduce-scatter sums matching data from every chip and leaves each chip holding just its own slice of the combined result.

RDMA lets one machine read or write the memory of another directly over the network, without either CPU copying data along the way.

ROCm is AMD’s software toolbox for running AI and other high-performance programs on AMD chips.

On InferenceX a roofline is the outer envelope drawn through the best points of one hardware configuration, showing the edge of what it achieved.

Scale-up is the ultra-fast network inside one tightly connected chip system. Scale-out is the broader network connecting separate servers or racks.

Sequence parallelism splits a single long sequence across chips, so the tokens of one request are processed by several accelerators at once.

An SLO is the performance promise a deployment has to keep, such as a first token within one second for nine requests in ten.

SGLang is open-source software for serving language models quickly, with scheduling and optimization features for complex AI workloads.

Sliding window attention lets a layer look only at a recent span of tokens, so its cache stops growing once the window is full.

Sparse attention lets the model look back at only the most useful parts of a long context instead of rereading every earlier token.

Speculative decoding lets a cheaper helper draft several tokens ahead, then asks the full model to approve them together instead of generating each one separately.

A subagent is an additional agent started by a main agent to handle a smaller piece of the same task, sometimes at the same time as other work.

Tail latency describes the slowest requests rather than the typical one, because the unlucky few are what users actually notice.

Tensor parallelism splits one large calculation across several chips so they solve it together.

TensorRT-LLM is NVIDIA’s optimized software stack for getting high inference performance from NVIDIA chips.

TDP is the sustained power a chip is designed to draw and shed as heat, the headline wattage on every accelerator spec sheet.

Throughput is how much total work the system gets done each second across everyone using it.

TileRT is an inference runtime built for very fast single-user generation, compiling a model into one resident program instead of many separate kernel launches.

TPOT is the gap between each new piece of a streamed answer. Smaller gaps make the response feel faster and smoother.

TTFT is the “thinking…” pause between sending your prompt and seeing the first piece of the answer.

Tokenization splits text into the sub word units a model actually reads and writes, and every performance and price number is quoted in them.

Tokens per dollar asks how many tokens one dollar of infrastructure spend can produce under the cost basis named on the chart.

Tokens per megawatt asks how much AI output a data center can produce from a fixed amount of available power.

TCO covers the hardware purchase plus the cost of powering, cooling, networking, and operating it over time.

Trace replay recreates the timing and shape of recorded sessions so a benchmark sends requests like the original workload.

Triton is a Python-based language for writing custom accelerator kernels, letting ML engineers get near hand-tuned speed without writing low level code.

UALink is an open industry standard for the fast scale up links between accelerators in a rack, the ecosystem answer to NVLink.

An unofficial run is a benchmark run that has not been ingested into the published dataset but can still be drawn on top of the charts from its URL.

vLLM is open-source software that organizes requests and chip memory so language models can serve many users efficiently.

Warmup is the priming pass before measurement starts, so the run is scored on a system in steady state rather than one with an empty cache.

Weight-only quantization compresses just the stored model weights to low precision while the math still runs in higher precision formats.

Wide expert parallelism spreads a model’s specialists across many chips, giving each chip less expert data to hold and move.