Agentic inference
Also known as AI agent inference, agent inference
In plain English
Agentic inference serves an AI system that works through a task over many model requests, often using tools and delegating work along the way.
Technical definition
Agentic inference is model serving for agents that maintain state across multiple turns, call tools, reuse growing context, and may run subagents in parallel.
Engineering details
A single agent session can alternate between model requests, tool execution, and waiting periods. Later requests often include much of the earlier conversation, so prefix caching and KV-cache capacity affect both speed and cost. Parallel subagents add branches with their own request timing and context growth.
Why it matters
Fixed input and output lengths miss several pressures created by agents. Long shared prefixes change cache behavior, tool delays make traffic bursty, and concurrent branches compete for serving capacity. Hardware and software can rank differently under this request pattern.
How to read it in InferenceX
InferenceX uses AgentX to measure agentic inference. Read AgentX results alongside fixed-sequence scenarios because they answer different capacity questions. AgentX reports the behavior of a closed-loop session replay instead of treating every request as an independent batch item.
Source material
See the concept in real benchmarks
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM