Vector Processing Unit
Also known as VPU, TPU vector unit, vector unit
In plain English
The VPU is the part of a TPU that handles elementwise math like activations, softmax, and state updates, working alongside the matrix unit.
Technical definition
The Vector Processing Unit is the SIMD engine inside a TPU TensorCore that performs elementwise, reduction, and data-rearrangement operations on tiles whose last dimension is 128 lanes wide.
Engineering details
The MXU handles matrix multiplies and the VPU handles everything around them: activation functions, normalization, softmax, rank-one state updates, and the scaled-vector additions that stitch kernel stages together. The VPU works on tiles with a 128-lane trailing dimension and 8 sublanes, so arrays with a smaller trailing dimension get padded up. Kernels get faster when MXU and VPU work overlap instead of waiting on each other, and slower when too many live values force vector-register spills to VMEM.
Why it matters
The Gated DeltaNet output-projection rewrite exists purely to overlap the two units. Previously the VPU applied the rank-one update and the MXU then multiplied the updated state by the query, so the MXU waited. Rearranging the algebra lets the MXU compute Sq from the decayed state while the VPU builds the next state, with the update contribution added as a per-head scalar times a vector.
How to read it in InferenceX
That overlap gave 8k1k throughput gains of 2.79% at concurrency 64 and 4.48% at concurrency 512 on Qwen3.5 397B. A follow-up sliced Q and K inside the decode loop to reduce vector-register spills, making the decode-64 kernel about 20% faster with end-to-end gains of 0.8% on 8k1k and 3.8% on 1k8k at concurrency 512.