The language behind the inference curve.

Acceptance length is how many drafted tokens the full model actually approves per verification step, which is what decides whether speculation pays off.

Active parameters are the weights a mixture of experts model actually uses for each token, a small slice of its much larger total size.

This is the request pattern created when a coding agent reads a repository, edits code, runs tools, and revisits the model until the task is done.

Agentic inference serves an AI system that works through a task over many model requests, often using tools and delegating work along the way.

AgentX is the InferenceX workload for testing how inference systems serve complete long-context, multi-turn coding-agent sessions.

Aggregated serving runs prefill and decode on the same set of chips, the default deployment shape before an engine adds prefill-decode disaggregation.

You give a trained model something new, such as a prompt, image, or audio. It uses what it learned to produce an answer.

AIPerf is the vendor-neutral client that sends the benchmark traffic, reconstructing recorded agent sessions and timing every request.

AITER is AMD’s tuned kernel library, the layer that decides how fast attention and matrix work actually run on Instinct chips.

All-gather is a group communication step where every chip ends up holding the combined data that started out split across all of them.

All-reduce lets every chip solve one piece of a calculation, combines those pieces, and gives the combined result back to everyone.

All-to-all is a coordinated exchange where every chip sends a different package of data to every other chip.

This estimate asks how much token sales could earn in a year from a fixed gigawatt of datacenter utility power.

An apples-to-apples comparison holds precision, serving mode, and workload constant so a difference in results can be attributed to the hardware.

Arithmetic intensity is how much math a computation does per byte it moves, which decides whether the chip or its memory is the limit.

ATOM is AMD’s own inference engine, its answer to a vendor runtime rather than an upstream open-source one.

Batching is like putting several passengers on one bus: the chip handles multiple requests together so each trip does more useful work.

BF16 is the 16-bit format most models are trained in, and serving in it is the accuracy reference the quantized recipes are measured against.

Billable utilization is the share of modeled serving capacity that is actually used for paid traffic over the period being estimated.

Block scaling gives each small group of low precision numbers its own shared scale factor, recovering range that tiny formats lack on their own.

Boardfly is the new flatter network in Google’s inference-only TPU 8i that cuts the number of hops between chips roughly in half compared with a torus.

A hot row is accessed often or recently enough that keeping it in a faster tier may save repeated retrievals.

Cached input tokens are parts of a prompt whose earlier processing can be reused, reducing the work needed to read the prompt again.

Chunked prefill reads a long prompt in slices instead of all at once, so other users keep receiving tokens while it is being read.

In a closed-loop benchmark, each simulated user waits for one step to finish before sending the next step in that session.

Concurrency is how many people or requests the system is serving at once.

Conditional memory supplies learned information only when the input selects the corresponding memory entries.

Context parallelism splits one long prompt across several chips so they share the work of reading it and of scanning its stored attention state.

The context window is the maximum number of tokens a model can consider at once, covering both the input and everything generated so far.

A gate controls how much a retrieved memory feature contributes in the current model context.

Continuous batching lets new requests join a running batch the moment old ones finish, instead of waiting for the whole batch to complete.

This is the estimated infrastructure bill for producing one million tokens, the chunks of text an AI model reads and writes.

CPU offloading keeps model data or reusable attention state in host DRAM, freeing accelerator memory while adding host-memory access costs.

CUDA is NVIDIA’s software toolbox for making programs run on its chips.

CUDA graphs record a whole sequence of chip operations once and replay it as a unit, removing the per-step cost of launching each one.

CUTLASS is an NVIDIA template library that provides the building blocks for writing matrix multiply kernels that approach peak hardware speed.

Data parallelism makes multiple copies of the model and divides incoming work among them, like opening more identical checkout lanes.

DP attention gives each rank its own slice of the attention work and its own cache, instead of every rank holding a copy of the same thing.

Decode is the model writing its answer one token at a time after it has read the prompt.

A dense model applies every one of its parameters to every token it processes, unlike sparse models that route tokens to a few experts.

A die-to-die link connects two compute dies inside the same chip package, faster than the network between chips but separate from a shared memory space.

Disaggregated inference gives prompt reading and answer writing to separate chip teams, so each team can be tuned for its own job.

Double buffering fetches the next chunk of data while the chip is still computing on the current one, so memory latency is hidden behind useful work.

A draft model is the small fast model in speculative decoding that guesses several upcoming tokens for the large model to verify in one pass.

DSX MaxLPS manages datacenter power around workload demand so operators can use capacity that conservative peak-power provisioning would leave unused.

This metric asks how fast a whole answer arrives, counting the wait before the first word as well as the streaming speed after it.

EAGLE is a particular way to draft several likely next tokens for the main model to check, which can make answers stream faster.

Embedding prefetch starts retrieving the rows a later model layer will need while earlier layers are still computing.

An embedding table stores learned vectors in rows that the model retrieves using input-derived identifiers.

End-to-end latency is the time from sending a model request until its complete answer has arrived, including the initial wait.

Energy per token is how much electricity the system spends to produce one token, the power-side counterpart of cost per token.

Engram retrieves learned vectors for recurring token patterns instead of rebuilding every pattern through the model layers.

Expert parallelism gives different chips different specialist parts of a model, then sends each token to the specialists it needs.

Co-design means developing connected parts of the system together so a gain in one part is not lost to a bottleneck elsewhere.

The filesystem page cache keeps recently accessed file data in RAM so later reads can avoid the storage device.

FlashAttention computes exact attention in fast on-chip memory tiles, avoiding the huge intermediate matrix that made attention slow and memory hungry.

FlashInfer is a library of attention kernels that serving engines call instead of writing their own attention implementations.

FP4 compresses model numbers into just 4 bits. That can make inference much faster and smaller, but leaves less room for numerical detail.

FP8 is a compact 8-bit way to store and calculate with model numbers, reducing memory use and often speeding up inference.

Functionalization rewrites code that mutates objects in place into code that takes state as input and returns new state, so a compiler can trace it.

Gated DeltaNet is a linear attention layer that keeps a fixed-size running state per request instead of a KV cache that grows with every token.

GEMM is the general matrix multiply operation, the single computation that consumes most of the arithmetic in training and serving neural networks.

Goodput counts only the work that meets your latency target, so a fast-looking system that misses deadlines gets no credit for it.

GPU utilization measures how busy an accelerator is, though the common percentage from monitoring tools can look high while real efficiency stays low.

A grouped GEMM runs many small matrix multiplies, one per expert, in a single launch so a mixture-of-experts layer does not pay per-expert overhead.

Grouped-query attention lets several query heads share one set of key and value heads, shrinking the KV cache without giving up much quality.

HBM is the chip’s small pool of extremely fast nearby memory, where model weights and working data must fit while inference runs.

A hybrid model mixes attention types across its layers, so its cache is several different kinds of state rather than one uniform block.

Hybrid prefix caching lets a model with both attention and linear-attention layers reuse a cached prompt by saving the recurrent state alongside the KV blocks.

The inference engine is the traffic controller behind an AI service: it keeps incoming requests moving and makes sure the chips do the right work at the right time.

An inference-time ablation disables part of an already trained model to measure how the model depends on that part.

InfiniBand is a high speed, low latency network fabric that connects servers in AI clusters, carrying traffic between nodes that NVLink cannot reach.

Input length is how much the model reads; output length is how much it writes. “8K/1K” means a long prompt followed by a shorter answer.

INT4 stores weights in four-bit integers, shrinking the model enough to move far less memory per token on hardware without native 4-bit floats.

INT8 stores numbers as 8 bit integers with a scale factor, halving memory versus 16 bit formats and doubling math rates on supporting hardware.

ICI is the custom network that lets TPU chips exchange data directly with each other across a whole pod without going through the host CPU.

Interactivity is how quickly one person sees new words appear after the model starts answering.

Internal TCO is what a chip costs its own designer to run; external TCO is what a customer pays to buy or rent it, and the two can differ a lot.

Iso-interactivity compares systems while users see words appear at the same speed. This provides an apples-to-apples view of the hardware behind the experience.

JAX is Google’s Python framework for array computing that compiles functions through XLA and has been the native way to program TPUs.

Kernel fusion merges several small chip operations into one, so intermediate data stays in fast memory instead of bouncing through HBM.

The KV cache is the model’s working memory for the current conversation. It keeps useful notes and avoids rereading everything for every new token.

P2P pooling joins the host memory of many servers into one logical KV cache store, so an accelerator on any server can pull cached context from any other.

A KV cache manager is the component that stores attention state outside the chips and decides what to keep, evict, and fetch back.

KV cache offloading parks attention state the chips cannot hold in host memory, so a long session can be resumed instead of recomputed.

This stores the conversation cache in a smaller format, so a chip holds more context and reads it back faster during generation.

KV head replication copies the same key-value heads onto several tensor-parallel ranks when a model has fewer KV heads than the devices it is split across.

KV-aware routing sends a request to the worker that already holds its conversation state, instead of to whichever worker is least busy.

Latency is how long you wait. For a streamed answer, that includes both the wait before it starts and the pauses between later words.

Linear attention keeps a fixed-size running summary instead of every past token, so its memory does not grow as the conversation does.

llm-d is a Kubernetes-native framework that runs vLLM across many nodes with smart routing, prefix-aware scheduling, and disaggregated prefill and decode.

Long context means the model is working over a very large amount of conversation, code, or documents at once, which stresses memory far more than raw compute.

The MXU is the grid of multiply-accumulate cells inside a TPU that does the matrix math, and it only runs at full speed when every cell has real work.

MegaCore was the TPU v4 and v5p design that fused two physical cores into one logical accelerator sharing a single memory space.

Memory bandwidth is the width of the pipe feeding data to the chip’s compute units. A wider pipe keeps them from sitting idle.

A workload is compute bound when the math units are the bottleneck and memory bound when waiting on data movement is the bottleneck.

A memory-mapped file exposes file contents through memory addresses while the operating system manages which pages are resident.

A mixture-of-experts model is like a large team of specialists: it calls only the few experts best suited to each token instead of using the whole team every time.

MFU compares the useful math a model actually performed against the maximum the chip could theoretically perform, giving an efficiency percentage.

This estimate subtracts modeled serving costs and applicable model-license fees from annual token revenue for a gigawatt of utility power.

Mooncake Store pools the host memory and NVMe drives across many servers into one shared KV-cache store that any accelerator in the cluster can read.

MLA compresses the model’s notes about earlier tokens so long conversations use less memory and are cheaper to continue.

MTP lets the model guess several upcoming tokens at once and then verify them, reducing the number of slow one-token-at-a-time steps.

A multi-turn session sends several related requests to the model, carrying earlier conversation and tool results into later prompts.

MXFP4 is a 4-bit format that gives small groups of numbers their own scale, helping very compact values keep enough useful range.

An n-gram embedding gives a short sequence of tokens a learned vector that the model can look up.

NCCL is the NVIDIA library that moves data between chips during collective operations, with RCCL as its AMD counterpart.

NVFP4 is NVIDIA’s Blackwell-optimized version of 4-bit model math, designed to move less data and use the chip’s fastest low-precision hardware.

NVIDIA Dynamo coordinates many chip workers. It routes requests, moves model memory, and assigns prompt reading and answer generation to the right pools.

NVL72 is a rack where 72 accelerators share one high-speed fabric, so they behave more like a single large machine than a cluster.

NVLink is NVIDIA’s high-speed highway between chips, allowing them to cooperate much faster than over ordinary server networking.

NVMe offloading stores model data or attention state on SSDs, trading cheaper capacity for a more expensive access path.

Offline inference processes a large pile of requests with no user waiting, so the only goal is maximum tokens per dollar, not latency.

An optical circuit switch uses tiny mirrors to redirect light between fibers, letting Google rewire a TPU pod without touching a cable.

P90 interactivity expresses a slower-tail streaming interval as a token rate, so higher values mean faster response streaming.

PagedAttention stores the KV cache in small fixed size blocks, like virtual memory pages, so cache memory is not wasted on unused space.

Pallas is the way engineers write hand-tuned kernels for TPUs when the compiler alone does not schedule an operation well enough.

Parameter offloading keeps learned model data outside accelerator memory and fetches the portions needed for computation.

The Pareto frontier is the line of best available tradeoffs. Each point remains viable because improving one dimension would require giving up ground on another.

Performance per dollar measures how much useful AI output the system produces for each dollar spent running it.

Pinned host memory stays resident in RAM so the device can use a stable host-memory allocation.

Pipeline parallelism gives each chip a different slice of the layers, passing activations along the chain instead of splitting each layer.

PJRT is the runtime interface that sits between a framework like JAX or TorchTPU and the TPU hardware, managing device memory buffers and execution.

PUE measures how much total datacenter power is consumed for every watt that actually reaches the computing equipment inside it.

Prefill is the model reading and understanding your prompt before it begins writing the answer.

The hit rate is the share of prompt tokens served from cache instead of being recomputed, which on long sessions is most of the prompt.

Prefix caching remembers the work for a repeated beginning, such as the same system prompt, so the model can skip that work next time.

PrivateUse1 is the PyTorch hook that lets a vendor plug in a new device type, so a tensor can live on device="tpu" just like it would on CUDA.

Quantization stores the model’s numbers with fewer bits, making it smaller and faster to move, usually with a carefully controlled loss of precision.

RadixAttention keeps completed KV cache in a radix tree keyed by token content, so any new request can reuse the longest matching prefix.

Ragged paged attention is the TPU attention kernel that handles a batch of requests with different lengths reading from a paged KV cache in one launch.

A reasoning model generates long hidden chains of thought before answering, trading extra output tokens for better results on hard problems.

A recipe is the complete set of choices behind one curve: which model, engine, image, precision, parallelism, and workload were run.

Recurrent state is the fixed-size memory a linear-attention or SSM layer keeps per request, unlike a KV cache that grows with every token.

Reduce-scatter sums matching data from every chip and leaves each chip holding just its own slice of the combined result.

RDMA lets one machine read or write the memory of another directly over the network, without either CPU copying data along the way.

ROCm is AMD’s software toolbox for running AI and other high-performance programs on AMD chips.

On InferenceX a roofline is the outer envelope drawn through the best points of one hardware configuration, showing the edge of what it achieved.

Scale-up is the ultra-fast network inside one tightly connected chip system. Scale-out is the broader network connecting separate servers or racks.

Sequence parallelism splits a single long sequence across chips, so the tokens of one request are processed by several accelerators at once.

Sequence-on-lane is a KV cache layout that puts a page of tokens along the 128-wide vector lane axis instead of the head dimension, so less memory is wasted on padding.

An SLO is the performance promise a deployment has to keep, such as a first token within one second for nine requests in ten.

SGLang is open-source software for serving language models quickly, with scheduling and optimization features for complex AI workloads.

SGLang-JAX is the JAX-native version of the SGLang serving engine for TPUs, separate from the PyTorch SGLang runtime.

Shape bucketing rounds each batch up to one of a few precompiled sizes so the compiler does not have to build a new program for every request count.

Sliding window attention lets a layer look only at a recent span of tokens, so its cache stops growing once the window is full.

Sparse attention lets the model look back at only the most useful parts of a long context instead of rereading every earlier token.

A sparse lookup reads a few selected embedding rows rather than reading the entire parameter table.

SparseCores are small helper cores on a TPU chip that handle irregular jobs such as gathers and data movement so the matrix units can keep multiplying.

Collective offload runs communication operations like all-reduce on the TPU SparseCore instead of the TensorCore, but only when the message is big enough to be worth it.

Speculative decoding lets a cheaper helper draft several tokens ahead, then asks the full model to approve them together instead of generating each one separately.

StableHLO is a standardized intermediate representation that frameworks like PyTorch and JAX emit so the XLA compiler can turn it into TPU machine code.

A subagent is an additional agent started by a main agent to handle a smaller piece of the same task, sometimes at the same time as other work.

An agent can launch several short-lived helpers together, suddenly adding requests and new context for the server to hold.

A systolic array is a grid of tiny multiply-add cells where data flows cell to cell like a heartbeat, so a matrix multiply never touches memory mid-computation.

Tail latency describes the slowest requests rather than the typical one, because the unlucky few are what users actually notice.

Teacher forcing supplies the reference token history so different model variants are scored on the same continuation.

Tensor parallelism splits one large calculation across several chips so they solve it together.

A TPU is the accelerator Google designs for its own AI workloads and now sells or rents to other companies for inference.

A TensorCore is the main compute block inside a TPU chip, holding the matrix units, vector units, and on-chip memory that run dense model math.

TensorRT-LLM is NVIDIA’s optimized software stack for getting high inference performance from NVIDIA chips.

TDP is the sustained power a chip is designed to draw and shed as heat, the headline wattage on every accelerator spec sheet.

Throughput is how much total work the system gets done each second across everyone using it.

Tile padding is the wasted work a chip does when a matrix dimension is smaller than its hardware tile and gets filled with zeros to fit.

TileRT is an inference runtime built for very fast single-user generation, compiling a model into one resident program instead of many separate kernel launches.

TPOT is the gap between each new piece of a streamed answer. Smaller gaps make the response feel faster and smoother.

TTFT is the “thinking…” pause between sending your prompt and seeing the first piece of the answer.

Tokenization splits text into the sub word units a model actually reads and writes, and every performance and price number is quoted in them.

Tokens per dollar asks how many tokens one dollar of infrastructure spend can produce under the cost basis named on the chart.

Tokens per megawatt asks how much AI output a data center can produce from a fixed amount of available power.

torch.compile is the PyTorch entry point that captures a model into a graph and hands it to a compiler backend instead of running each operation one at a time.

TorchAX let PyTorch model code run on TPUs by secretly translating each operation into JAX, an approach TorchTPU is now replacing.

TorchTPU makes a TPU look like an ordinary PyTorch device, so vLLM and SGLang can run their existing PyTorch code on Google chips.

A torus is a grid network whose edges wrap around and connect to each other, so the farthest chip is never more than half the grid away.

TCO covers the hardware purchase plus the cost of powering, cooling, networking, and operating it over time.

A superpod is the largest unit of TPUs wired into one ICI network, up to 9,216 Ironwood chips that can be treated as a single machine.

TPU-Sync is Google’s library for moving KV cache between TPUs and out to host memory, the plumbing that disaggregated serving and offloading need.

Ironwood is the seventh-generation TPU and the first one Google sells for other companies to run their own inference on.

Trace replay recreates the timing and shape of recorded sessions so a benchmark sends requests like the original workload.

Triple buffering keeps three data blocks in flight at once, one being computed and two being fetched, so the compute unit rarely waits on memory.

Triton is a Python-based language for writing custom accelerator kernels, letting ML engineers get near hand-tuned speed without writing low level code.

A twisted torus is a torus network whose wraparound links are shifted, like a Mobius strip, so the average number of hops between chips drops further.

UALink is an open industry standard for the fast scale up links between accelerators in a rack, the ecosystem answer to NVLink.

UVA lets supported host and device allocations share an address space that GPU code can use.

An unofficial run is a benchmark run that has not been ingested into the published dataset but can still be drawn on top of the charts from its URL.

A utility power budget is the electricity capacity available to the whole datacenter, including the equipment that powers and cools the servers.

VMEM is the fast on-chip scratch memory a TPU kernel works from, and how much of it a kernel uses decides how far ahead it can prefetch.

The VPU is the part of a TPU that handles elementwise math like activations, softmax, and state updates, working alongside the matrix unit.

Vera Rubin is NVIDIA’s platform combining Rubin GPUs, Vera CPUs, and the interconnect and networking components around them.

vLLM is open-source software that organizes requests and chip memory so language models can serve many users efficiently.

Warmup is the priming pass before measurement starts, so the run is scored on a system in steady state rather than one with an empty cache.

Weight-only quantization compresses just the stored model weights to low precision while the math still runs in higher precision formats.

Wide expert parallelism spreads a model’s specialists across many chips, giving each chip less expert data to hold and move.

XLA is the compiler that turns a model graph into TPU machine code, deciding how to tile, pad, fuse, and schedule every operation.