Software
Kernel launch overhead
Also known as launch latency, kernel dispatch overhead, launch-bound execution
In plain English
Kernel launch overhead is the work needed to dispatch device operations; when operations are short, dispatch gaps can take a noticeable share of a decode step.
Technical definition
Kernel launch overhead is the host and runtime cost of submitting device work, together with dispatch-related delays that can leave gaps between useful device operations.
Engineering details
A CPU-side launch duration is not automatically equal to accelerator idle time because dispatch and execution may overlap. A timeline distinguishes submission work, synchronization, dependency stalls, and actual device gaps. Graph replay and fusion reduce different parts of this overhead. Neither technique eliminates the need to launch work or makes memory transfers and collectives free.
Why it matters
Low-batch decode can be launch-bound even when peak compute capability is high. Counting launches alone cannot establish speedup: the remaining kernels may take longer, and the critical path can move to memory or communication.
How to read it in InferenceX
The TPU article contrasts conventional decode with megakernels and cites an AMD reduction in launch count. Treat that count and the reported speedup as properties of the named model and recipe, not as a universal conversion between launches and tokens per second.
See the concept in real benchmarks
All articlesThe Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
TPUv7 AgentX results across Muse Spark Open, GLM, Qwen3.5, and Kimi K3, with TorchTPU, megakernels, and llm-d disaggregation.
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine