llm-d
Also known as llm-d project, Kubernetes LLM serving, distributed inference orchestrator
In plain English
llm-d is a Kubernetes-native framework that runs vLLM across many nodes with smart routing, prefix-aware scheduling, and disaggregated prefill and decode.
Technical definition
llm-d is an open-source distributed inference serving framework built on Kubernetes and vLLM that provides KV-aware routing, prefill-decode disaggregation, and multi-node scheduling for large model deployments.
Engineering details
A single vLLM instance serves one replica. Production deployments need a layer that routes each request to the replica most likely to hold its prefix in cache, splits prefill and decode across separate pools, and scales those pools independently. llm-d supplies that orchestration, backed by Red Hat, Google, and other contributors. TPU support in llm-d is part of Google’s externalization work for prefill-decode disaggregation, alongside the TPU-Sync transfer library, so external TPU customers can run the same disaggregated topology Google uses internally for Gemini.
Why it matters
Disaggregation and KV-aware routing are where much of the remaining performance-per-dollar gain sits for TPU, and they only exist above the engine. Shipping TPU support in a widely used orchestrator matters as much as the engine backend for making those features usable outside Google.
How to read it in InferenceX
InferenceX single-node aggregated results do not exercise llm-d. The planned TPUv7 disaggregated comparison against GB200 and GB300 NVL72 depends on the llm-d and TPU-Sync work landing in the external stack.
Source material
See the concept in real benchmarks
TPU Inference Externalization Full Steam Ahead
InferenceX, Up to 50% Better Performance per Dollar, Rapid Externalization of TPU stack, Growing Customer Base, Ironwood, TPUv8i, Reducing CUDA Moat
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure