Matrix Multiply Unit
Also known as MXU, systolic array, TPU matrix unit
In plain English
The MXU is the grid of multiply-accumulate cells inside a TPU that does the matrix math, and it only runs at full speed when every cell has real work.
Technical definition
The Matrix Multiply Unit is the systolic array at the center of a TPU TensorCore, a two-dimensional grid of multiply-accumulate cells that holds weights stationary while activations stream through.
Engineering details
Weights load into the array and stay put. Activations enter from one edge, partial sums ripple across the grid cell by cell, and the finished result streams out the far side without touching memory mid-computation. TPU generations through v5 used a 128x128 array, or 16,384 MACs per cycle. From v6e onward, including Ironwood, the array is 256x256, or 65,536 MACs per cycle. The XLA compiler pads any matrix dimension smaller than the array side up to fill the tile, and every padded cell still consumes a MAC that cycle.
Why it matters
A wider array is only free when it stays full. A head dimension of 128 on a 256-wide MXU caps the attention matmuls at 50% utilization, and a head dimension of 64 caps them at 25% before any kernel code is written. GPU matrix cores consume small tiles, so the same shapes cost almost nothing on an H100 or B200. Model hyperparameters that were arbitrary on GPUs become a direct throughput tax on TPUs.
How to read it in InferenceX
Bring-up cost on TPU correlates with how well a model fits the MXU rather than with how popular the model is. Qwen3.5 was chosen as the first TorchTPU bring-up model partly because its shapes fall out cleanly, and the Pallas kernel work described in the InferenceX Official Preview centers on keeping the array fed.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
OpenAI Jalapeño: Better Than Nvidia Blackwell
OpenAI’s self-designed ASIC compared with Rubin, Jalapeño’s TCO, throughput per MW, and spicy deets