Tile padding
Also known as shape padding, MXU padding, lane padding
In plain English
Tile padding is the wasted work a chip does when a matrix dimension is smaller than its hardware tile and gets filled with zeros to fit.
Technical definition
Tile padding is the rounding up of a tensor dimension to the native tile size of a compute or vector unit, with the padded cells occupying hardware cycles while contributing nothing to the result.
Engineering details
On TPU the MXU side length is 128 on older generations and 256 on v6e and v7, and the vector unit works on tiles whose trailing dimension is 128 lanes. XLA pads any smaller axis to fill the tile. Llama 3 8B has a head dimension of 128, exactly half of a 256-wide MXU, which caps its two attention matmuls at 50% utilization. gpt-oss ships head dimension 64 and caps at 25%. DeepSeek MLA splits its query-key dimension into 128 plus 64 for 192, awkward against any power-of-two array. In the batched attention kernel, a single FP8 KV head per device left half of every tile as padding until a sequence-on-lane layout put tokens on the 128-lane axis instead.
Why it matters
GPU matrix cores consume small tiles, so head dimensions of 64 or 128 and odd post-sharding KV head counts land near peak on H100 or B200. On TPU the same choices are a direct throughput tax before any kernel is written. Bring-up cost therefore tracks how well a model fits the tile geometry rather than how popular it is.
How to read it in InferenceX
The sequence-on-lane KV layout doubled usable KV pages from 5,141 to 10,283 in the reported Ironwood configuration and lifted 8k1k throughput 16.5% at concurrency 128 while cutting median TTFT by 95%. Explicit packing dimensions in ragged paged attention exist to work around XLA default tiling.