Offline inference
Also known as batch inference, offline batch serving
In plain English
Offline inference processes a large pile of requests with no user waiting, so the only goal is maximum tokens per dollar, not latency.
Technical definition
Offline inference is model serving without an interactive deadline, where requests are processed in bulk and the objective is total throughput and cost rather than per-user latency.
Engineering details
Synthetic data generation, document processing, embedding backfills, and evaluation sweeps do not care when any individual request returns. That frees the scheduler to run the largest batches the memory allows, order requests to maximize prefix reuse, and hold the accelerator at its throughput limit. Online serving lives at the opposite end of the same tradeoff, sacrificing throughput to keep every user above an interactivity floor.
Why it matters
The same hardware can differ by multiples in tokens per dollar between offline and tight latency operation, so quoting a single price per million tokens without stating the operating point is close to meaningless. Fleets often split into latency tiers for this reason.
How to read it in InferenceX
The right edge of an InferenceX throughput versus interactivity curve, where batch size is maximal and per-user speed is lowest, approximates offline operation. Reading one curve at both edges shows the full online to offline cost range for a recipe.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
B200 NVFP4 vs H100 FP8 on MiniMax-M2.5: Up to 8.2x Better Performance per Dollar with vLLM
vLLM PR #36307 unlocks the trtllm-gen FP8 MoE kernel for MiniMax on B200; combined with NVFP4, perf/$ scales from 4.0x at 22 tok/s/user to 8.2x at 110 on 8K/1K