SGLang

HiCache is SGLang’s first-class in-tree offloading mechanism. It faced the same hybrid problem vLLM’s connectors did, and solved it with an asymmetry: offload the full-attention cache and reconstruct the short sliding-window tail on the way back. Only the expensive half is worth moving across the bus, and the cheap half can be rebuilt faster than it can be fetched. On AMD, staged write-back keeps that movement from blocking the engine while it happens.

Recurrent state was the remaining gap, because it cannot be rebuilt from neighbouring tokens the way a window tail can. FlashInfer GDN checkpoints let it participate in prefix reuse at all, and raised throughput from 47,771 to 53,004 tok/s/GPU at a 92.4% cache-hit rate.

Like production traffic, AgentX sessions arrive at continuously varying context lengths. A naive runtime that specializes on length will compile a fresh kernel for nearly every request it sees. SGLang maintainers solved this by passing context length as a runtime scalar, collapsing that into one compilation, and improved AgentX output throughput at concurrency 384 by 26.75% and mean TTFT by 36.25% — by removing compilation, not by computing anything faster.

In the same spirit, removing a per-step device-to-host sequence-length synchronization eliminates a decode bubble that exists only because the host wanted to know a length the device already had.

When a request carries no reusable history, such as the start of a subagent, any worker will do and load balancing is the only question worth asking. When it carries a megabyte of cached prefix, sending it to an idle worker that does not hold that prefix is the expensive choice, and the router needs to know where the state already lives.

SGLang added DP cache affinity so a session is sticky to the rank holding its cache. The same work implements DP-aware prefill and decode routing so both halves of a disaggregated deployment make that decision consistently, and adds cache balance as a routing signal so affinity does not degenerate into one hot worker. A router can only act on what it is told, so hybrid cache events also became radix-cache aware and sliding-window aware.

Speculative decoding receives special attention, because MTP adds a second, smaller piece of per-request state that has to survive everything the main cache survives. SGLang fixed draft-window transfer in disaggregated serving so that state crosses the prefill-to-decode boundary intact, added overlap scheduling for high-concurrency online decoding, removed a no-op EAGLE renormalization, and avoided host synchronizations during EAGLE prefill.

The open resource-lease scheduling work and the data-parallel graph-metadata fix continue the same effort: making overlap safe when requests can be retracted and resumed rather than simply run to completion.

Heterogeneous prefill and decode topologies need prefix-aware staging, and this is where prefix caching and disaggregation interact badly. When the two sides are not sharded identically, KV cannot be copied across as one contiguous stream; it has to be split on a transfer grid and reassembled at the offsets the decode side expects. A prefix hit makes that harder, not easier, because the prefill worker now sends only the uncached remainder while the decode side still expects a complete, correctly positioned cache. Radix-cache support in the staging buffer splits cached sends on that grid and scatters them at the correct decode offsets.

The failure this fixes is a correctness failure. A 127,500-token shared-prefix test went from 2 correct needles out of 128 to 128 out of 128, meaning the cache had been silently landing in the wrong places — which a throughput benchmark would have scored as a fast, confident, wrong answer. The AgentX comparison additionally raised median per-user output throughput by 9.61% at nearly unchanged total throughput per GPU.

Open work continues along the same seam: multi-pool DeepSeek-V4 support in UMBP, unified-KV HiSparse state carried over MoRI, and preserving the prefill-owned token when decode terminates without visible content. The HiSparse work should be read as a capacity and correctness enabler for long contexts rather than as a throughput win at high concurrency, which it is not yet.