Software
Decode megakernel
Also known as megakernel, whole-model kernel, MonoKernel
In plain English
A decode megakernel combines many operations across model layers into a long-running device program, reducing gaps between launches while overlapping weight transfers with computation.
Technical definition
A decode megakernel fuses a substantial part of the model forward pass across operations or layers, with explicit scheduling of computation, data movement, and sometimes collectives.
Engineering details
Conventional execution launches many separate kernels for attention, projections, expert routing, and other operations. A megakernel can prefetch the next layer’s weights while computing the current layer. Implementations differ in how much they fuse; the name does not guarantee that every operation, communication step, or request runs in one launch. Engine scheduling and prefix-cache management can remain outside the fused forward path.
Why it matters
At low concurrency, short operations may leave the accelerator idle between launches. Reducing those gaps can improve interactivity, but long-context attention, prefill, cache transfers, and high-concurrency throughput still need separate measurement.
How to read it in InferenceX
The November 2026 TPU AgentX article describes SGLang and vLLM megakernel paths and AMD’s MonoKernel. Inferact integrates its fused forward path with vLLM scheduling and KV transfer; these results do not imply that the full serving system is a single kernel.
See the concept in real benchmarks
All articlesThe Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
TPUv7 AgentX results across Muse Spark Open, GLM, Qwen3.5, and Kimi K3, with TorchTPU, megakernels, and llm-d disaggregation.
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine