Double buffering
Also known as triple buffering, prefetch pipelining, DMA overlap
In plain English
Double buffering fetches the next chunk of data while the chip is still computing on the current one, so memory latency is hidden behind useful work.
Technical definition
Double buffering is a kernel technique that allocates two on-chip buffers so a DMA transfer into one overlaps computation on the other, with triple buffering extending the same idea to a deeper pipeline.
Engineering details
On TPU a Pallas kernel hides HBM latency by fetching the next block into VMEM while the MXU works on the current block. Both blocks must fit in VMEM, so compute block size sets how far ahead the prefetch can run. The SparseCore ReduceScatter uses double buffering to transfer one chunk while accumulating another, overlapping local reductions and die-to-die transfers with slower chip-to-chip traffic. The grouped matmul triple-buffers expert weights so the next group’s weights are in flight while the current group computes. Asynchronous GDN state transfers use the same pattern, which initially cost VMEM until scratch buffers were reused.
Why it matters
Much of the reported TPU kernel gain comes from overlap rather than from faster arithmetic. Getting the buffer budget wrong shows up as either exposed memory latency or VMEM regressions elsewhere in the kernel.
How to read it in InferenceX
Splitting the ragged paged attention compute block from the fetch block gave the prefetch pipeline room and raised decode throughput 49% on Qwen3-0.6B. Asynchronous GDN state transfers gained 11.3% on 8k1k at concurrency 512 once VMEM was recovered. Batched ragged paged attention triple-buffers to reduce padding and improve pipelining.