Service level objective
Also known as latency target, SLA target
In plain English
An SLO is the performance promise a deployment has to keep, such as a first token within one second for nine requests in ten.
Technical definition
A service level objective is a stated target for a serving metric, usually expressed as a percentile bound on latency or interactivity.
Engineering details
A useful SLO names a metric, a percentile, and a threshold together. Serving capacity is then whatever throughput the system sustains without breaching it, which is a smaller number than peak throughput and the only one an operator can safely provision against.
Why it matters
Every point on a throughput curve is reachable, but only part of the curve satisfies a given promise. Two systems can look close on peak throughput and differ sharply in how much of that throughput survives an interactivity or first-token bound.
How to read it in InferenceX
InferenceX does not impose one industry SLO, because acceptable targets differ by product: interactive coding needs a high token rate, while batch processing tolerates seconds of first-token delay. Read the frontier at your own threshold instead of comparing peak values.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost