SparseCore collective offload
Also known as collective offload, SparseCore offload threshold, SC collective
In plain English
Collective offload runs communication operations like all-reduce on the TPU SparseCore instead of the TensorCore, but only when the message is big enough to be worth it.
Technical definition
SparseCore collective offload is the Ironwood technique of executing all-reduce, all-gather, and reduce-scatter collectives on SparseCores so the TensorCore stays free for matrix work, gated by a size threshold below which the collective stays on the TensorCore.
Engineering details
SparseCores are built for irregular data movement, which makes them a natural home for collectives. Google implemented ReduceScatter on SparseCore with a two-level scheme that reduces within a chip over the die-to-die link before exchanging across ICI, using double buffering to overlap stages. Offload is not free: launching on SparseCore has overhead, and for small messages that fit in VMEM the TensorCore finishes faster. For Qwen3.5 a threshold derived from VMEM capacity decides per collective which unit runs it.
Why it matters
The threshold is a reminder that moving work off the critical unit can backfire. In some cases offloading a collective to SparseCore worsens performance, so the gain comes from choosing per message size rather than from offload as a blanket policy. It also shows how TPU serving optimization touches scheduling decisions that GPU stacks leave to NCCL.
How to read it in InferenceX
The threshold-based offload gave Qwen3.5 397B 8k1k throughput gains of 2.7% at concurrency 64 and 5.7% at concurrency 128. Merging two separate all-gathers of expert IDs and routing weights into one saved about 80 microseconds per layer, or roughly 4.64 milliseconds per forward pass over DeepSeek-V3 style 58 layers.