Multi-turn inference
Also known as multi-turn serving, multi-turn workload
In plain English
A multi-turn session sends several related requests to the model, carrying earlier conversation and tool results into later prompts.
Technical definition
Multi-turn inference serves a sequence of related model requests whose inputs include state or conversation history from earlier turns.
Engineering details
In agentic workloads, a session may span tens or hundreds of turns. Generated output and tool results extend the next prompt, so much of its input may already have cached state. Tool execution and dependent requests also change when work reaches the server.
Why it matters
A sequence of independent prompts does not reproduce these dependencies or the growing cache working set. Cache eviction can force repeated prefill, while a slow response can delay the next turn even when aggregate throughput appears high.
How to read it in InferenceX
The Rubin article identifies multi-turn structure as a defining AgentX workload property. Compare these results with the same agentic scenario rather than transferring rankings from fixed-length, independent-request tests.
Source material
See the concept in real benchmarks
Rubin NVL72 Agentic Inference: 67x better Performance per Dollar
Jensen Sandbagging Performance Again, 2x more Annual Profit Per GigaWatt, The More you Buy, The More you Earn, AgentX, InferenceX, Extreme Co-Design
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200