Triple buffering
Also known as triple buffering, three-stage prefetch, deeper pipelining
In plain English
Triple buffering keeps three data blocks in flight at once, one being computed and two being fetched, so the compute unit rarely waits on memory.
Technical definition
Triple buffering is a kernel pipelining technique that allocates three on-chip buffers so the next two blocks of data can be prefetched from HBM while the current block is processed, extending double buffering by one stage.
Engineering details
Double buffering hides HBM latency by fetching block N+1 while computing block N. Triple buffering adds block N+2, which helps when transfer time varies or when the fetch of one block is longer than the compute on another. The cost is VMEM: three buffers must fit alongside everything else the kernel keeps live. On Ironwood the second version of the grouped matmul triple-buffers expert weights so the next expert group is already in flight while the current one is computed, and batched ragged paged attention triple-buffers page data to reduce padding and improve pipelining.
Why it matters
Deeper pipelining trades scarce VMEM for MXU utilization. The choice depends on block size, which is why the RPA v3 fix separated the KV fetch block from the KV compute block: shrinking compute to 4k tokens while keeping fetch at 16k freed room for the prefetch buffers.
How to read it in InferenceX
The grouped matmul v2 with triple-buffered expert weights is part of the MoE kernel work that fed the Qwen3.5 397B results in the TPU InferenceX preview. The batched RPA change was measured on Qwen3-32B, so its numbers are kept separate from the 397B comparison.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM