Shape bucketing
Also known as compiled shape buckets, padding buckets, token buckets
In plain English
Shape bucketing rounds each batch up to one of a few precompiled sizes so the compiler does not have to build a new program for every request count.
Technical definition
Shape bucketing is the practice of compiling a kernel or graph for a fixed set of tensor shapes and padding each live batch up to the nearest bucket, trading some wasted work for avoiding recompilation.
Engineering details
Ahead-of-time compilers such as XLA specialize on static shapes, so a serving engine cannot compile a fresh program for every combination of request count and sequence length. It picks buckets instead. When the buckets are tuned for hundreds of concurrent requests and only four or eight are in flight, metadata is sized for the configured maximum, padding tokens trigger dummy expert work, and scheduling overhead dominates. The TPU stack now buckets request metadata by active request count, adds a dedicated attention bucket for concurrency four, reduces the minimum token bucket, and routes padding tokens to expert zero so they do not load extra expert weights.
Why it matters
InferenceX operating points sit at exactly these low concurrencies, so bucket tuning directly changes reported curves. The same issue appears on GPUs as CUDA graph capture sizes; the TPU version is stricter because XLA compiles whole graphs per shape.
How to read it in InferenceX
Bucketing metadata by active requests cut GDN scheduling overhead from 283 to 97 microseconds on 8k1k at concurrency 64 and raised throughput from 2,328 to 2,516 tokens per chip per second. Qwen3.5 InferenceX tuning gained 13.3% on 8k1k and 15.5% on 1k1k at concurrency four, and a further round gained 22.9% on 1k1k at concurrency four.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200