Systolic array
Also known as systolic array, weight-stationary array, systolic matrix unit
In plain English
A systolic array is a grid of tiny multiply-add cells where data flows cell to cell like a heartbeat, so a matrix multiply never touches memory mid-computation.
Technical definition
A systolic array is a two-dimensional grid of multiply-accumulate cells in which one operand is held stationary while the other streams through, with partial sums passed between neighbors on every clock cycle.
Engineering details
In a TPU MXU, weights load into the array and stay put. Activations enter from one edge, each cell multiplies its input by its stored weight and adds the partial sum arriving from its neighbor, and the finished result exits the far side. No intermediate value is written to memory. The array has a fixed side length, 128 on TPU generations through v5 and 256 from v6e onward, and any matrix dimension smaller than that is padded to fill it. GPU tensor cores instead consume small tiles fed from registers, which is why they tolerate odd shapes better.
Why it matters
A systolic design gives very high MACs per cycle per unit of silicon and power, which is part of why Google can price Ironwood below Blackwell per token. The trade is rigidity: the 256x256 Ironwood array delivers 65,536 MACs per cycle only when both matrix dimensions are multiples of 256, and a head dimension of 128 caps attention matmuls at 50% utilization.
How to read it in InferenceX
The TPU InferenceX preview and the OpenAI Jalapeno analysis both treat systolic-array geometry as the main reason bring-up cost varies by model. Models whose shapes fill the array need weeks of tuning; models that fight it need new Pallas kernels before reaching parity.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
OpenAI Jalapeño: Better Than Nvidia Blackwell
OpenAI’s self-designed ASIC compared with Rubin, Jalapeño’s TCO, throughput per MW, and spicy deets