Aggregated serving
Also known as agg serving, colocated prefill and decode, non-disaggregated serving
In plain English
Aggregated serving runs prefill and decode on the same set of chips, the default deployment shape before an engine adds prefill-decode disaggregation.
Technical definition
Aggregated serving is the deployment mode in which each replica handles both the prefill and decode phases of every request on the same accelerators, as opposed to disaggregated serving with separate prefill and decode pools.
Engineering details
In aggregated mode a scheduler interleaves prompt processing and token generation on one set of devices, usually with chunked prefill so long prompts do not stall decode. It needs no KV-cache transfer between pools and is simpler to operate, but prefill and decode compete for the same compute and memory bandwidth, and neither phase can be scaled or tuned independently. Disaggregation separates them and typically wins at high concurrency on rack-scale systems, at the cost of a fast transfer path such as NIXL on NVIDIA or TPU-Sync on TPU. The external TPU stack currently serves in aggregated mode only.
Why it matters
Comparing aggregated against disaggregated is an apples-to-bananas comparison: it mixes a deployment-mode difference into a hardware comparison. InferenceX labels the mode on every curve so readers can separate the two.
How to read it in InferenceX
The InferenceX Official Preview compares TPUv7 aggregated FP8 serving against B200 and B300 aggregated FP8 serving, where Ironwood reaches up to 50% better performance per dollar. Against GB300 NVL72 disaggregated serving, TPUv7 aggregated is competitive at low and high end-to-end latency but trails by about 30% in the middle of the curve until TPU disaggregation is optimized.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.