Tensor Processing Unit
Also known as TPU, Google TPU, Cloud TPU
In plain English
A TPU is the accelerator Google designs for its own AI workloads and now sells or rents to other companies for inference.
Technical definition
A Tensor Processing Unit is a Google-designed accelerator built around large systolic matrix units, an on-chip vector memory, SparseCores for irregular work, and a chip-to-chip fabric called ICI.
Engineering details
TPUs run Search, Ads, YouTube, and every Gemini generation inside Google. Each chip pairs TensorCores, whose MXU systolic arrays do the matrix multiplies, with SparseCores that handle embedding lookups and data movement. The XLA compiler schedules work across those units, and Pallas kernels cover operations the compiler does not tile well. Chips connect through ICI into pods of thousands of devices without passing through a host CPU. Ironwood (TPUv7) is the first generation Google offers for outside inference customers, and the TPUv8 lineup splits into 8t for training and 8i for inference.
Why it matters
The design trades single-chip peak for system-level cost. Wide matrix units, a low-latency torus, and compiler-driven scheduling give Google a lower cost per token than a spec sheet would predict, but tile geometry punishes model shapes that do not fill the array. External adoption depends on the PyTorch-native TorchTPU stack reaching parity with the CUDA ecosystem in vLLM and SGLang.
How to read it in InferenceX
The InferenceX Official Preview publishes the first third-party TPUv7 results, comparing FP8 aggregated serving against B200 and B300. Ironwood reaches up to 50% better performance per dollar in that comparison, and AgentX and disaggregated TPU results follow once the TorchTPU stack is open sourced.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton