Kernel fusion
Also known as fused kernels, operator fusion
In plain English
Kernel fusion merges several small chip operations into one, so intermediate data stays in fast memory instead of bouncing through HBM.
Technical definition
Kernel fusion combines multiple consecutive operations into a single kernel launch, keeping intermediate results in registers or on-chip memory rather than writing them to main memory between steps.
Engineering details
An unfused sequence like matrix multiply, bias add, and activation writes its intermediate tensor to HBM after each step and reads it back for the next. Fusing them into one kernel eliminates those round trips and the launch overhead between them. Fusion happens by hand in libraries, through template epilogues in CUTLASS style code, and automatically in compilers such as torch.compile and Triton based stacks. FlashAttention is the most famous single example of the idea.
Why it matters
In memory bound inference, removing intermediate traffic is worth more than raw arithmetic improvements, so fusion is one of the most reliable levers engines have. A large share of version over version engine speedups reduces to more aggressive or better targeted fusion.
How to read it in InferenceX
InferenceX recipes pin engine images whose fusion inventories differ by version and by hardware backend, which is a recurring reason day over day tracking shows curves moving on unchanged silicon as fused kernels land for new models and precisions.
Source material
See the concept in real benchmarks
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool