CUDA graphs
Also known as graph capture, full-graph mode
In plain English
CUDA graphs record a whole sequence of chip operations once and replay it as a unit, removing the per-step cost of launching each one.
Technical definition
CUDA graph capture records a sequence of kernel launches and their dependencies into a replayable graph, so the sequence is submitted once rather than launched operation by operation.
Engineering details
A decode step issues many small kernels, and at small batch sizes the launch and scheduling overhead between them can rival the arithmetic. Capturing the step removes that per-launch cost. The catch is that a graph is fixed: shapes must be stable, so engines capture per bucket and leave genuinely dynamic work outside the graph.
Why it matters
This is a latency optimization more than a throughput one, and it matters most exactly where batches are small and interactivity is high. It also interacts with everything that changes shape, which is why variable-length agentic traffic can defeat a runtime that specializes too eagerly and recompiles for nearly every request it sees.
How to read it in InferenceX
Graph usage is part of the engine image a recipe pins, so it can move a curve with no change in hardware. Recipes may capture stable producers while leaving request-dependent attention eager, which is a deliberate compromise between capture coverage and shape flexibility.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM