AMD ATOM

Two earlier cache-manager fixes had to land before any of that could be measured, and both are worth noting as examples of a cache that reports itself healthy while doing nothing. One stopped free-pool hits from destroying shared cache entries. The other, a deferred-output fix, restored prefix hashing in the default scheduler mode and moved repeated long prompts from zero cached tokens to reuse of every complete prefix block.

A separate change lets prefix-hit prefill stay on the optimized sink attention kernel rather than falling back to the generic path, so a cache hit does not quietly cost part of what it saves.

Hybrid models also carry a recurrent or compressor state, which differs from ordinary KV in one decisive way: it cannot be reconstructed from the tokens around it. A window tail can be recomputed from neighbouring context, but recurrent state is the accumulated result of everything that came before, so if it is dropped the only way back is to replay the sequence.

ATOM gave this per-request state a content-addressed checkpoint lifecycle, letting generated turns leave reusable resume points without reserving a separate protected cache for them. In one test a request reused 512 generated tokens and computed only a two-token suffix.

The tuning detail matters as much as the feature. Publishing a checkpoint unconditionally cost 17.5% throughput on zero-hit traffic — the price paid by every session that never comes back, in order to help the ones that do. Spacing checkpoints by token interval avoided that penalty, and fixed 1k/1k throughput stayed within measurement noise, which is the relevant safety property: a feature aimed at agentic reuse should not tax workloads that will never use it.

ATOM’s AgentX-relevant CPU path starts from the arithmetic that justifies offload at all. Standalone LMCache offload reloads a 32,000-token prefix from CPU in about 0.32 seconds against roughly 2.5 seconds to recompute it, an eight-fold margin. That makes crossing the bus worth doing at these context lengths, and it would not hold for a short prompt.

The rest of the path is about ownership and index placement rather than bandwidth. ATOM copied vLLM’s multi-connector design, which lets a prefill worker send KV to a remote decode worker and save the same prefix to CPU at once, without freeing the blocks until both consumers are finished. Two independent readers of the same blocks is a situation single-turn traffic never produces.

Promoting restored blocks back into the GPU prefix index fixes a subtler waste: without it, a prefix loaded from CPU is used and then not registered as resident, so the next turn fetches the same hot prefix across the bus again, paying the transfer repeatedly for a cache that was already in HBM. Follow-up work fixed asynchronous save ordering, packed-KV geometry, unaligned handoffs, and remote request accounting together, eliminating reload corruption across a two-round, 2,638-request validation. That bug surfaces only when the same blocks are saved, evicted, and restored many times over.

The distributed path repeats, in a different codebase, the pattern already visible in SGLang and Dynamo: routing has to know where state lives. ATOM’s router, ATOM Mesh, is a fork of SGLang’s router with most features removed — including SGLang’s cache-aware routing, which it turned out to need.

ATOM gained KV lifecycle events for cache-aware routers so the router can know where state lives at all, multi-node prefill and decode routing, and session-sticky data-parallel routing. The sticky policy is a two-sided compromise worth stating explicitly: a conversation returns to the healthy worker that owns its state, but idle assignments expire so that stickiness does not permanently unbalance the cluster on behalf of sessions that have gone away.

Disaggregation has to move whatever the model actually keeps, which is not always one uniform cache. DeepSeek-V4 transfers both buffers of its mixed FP8 and BF16 cache layout, and EAGLE disaggregation moves the draft model’s independent KV cache alongside the target cache — the same second-cache problem TensorRT-LLM and SGLang each had to solve.

Remote-KV admission and backpressure closes the loop by stopping the decode side from accepting more parked transfers than it can safely resume, which is the disaggregated form of accepting work you cannot finish.

Long prefill receives parallelism that a fixed 8k prompt does not strongly exercise, because at 8k there is little to divide and TTFT is already short. Prefill context parallelism (PCP) splits DeepSeek-V4 query tokens across GPUs and reported 35 to 43% lower mean time to first token, with total throughput gains of up to about 49% at a 64,000-token input — a gain that grows with input length rather than with batch size.

Making that usable in practice required it to compose with everything else a session relies on, so decode context parallelism was made compatible with prefix caching, chunked prefill, and FP8 KV, and then extended to MTP. Parallelism that cannot coexist with the prefix cache would trade one long-context win for another.

Chunked pipeline-parallel prefill attacks the same problem from the memory side, replacing repeated tensor-parallel collectives with streamed layer-stage handoffs. Its GLM-5.2 result at high load is the most complete in this section: output throughput doubled, median time to first token fell from 28.6 seconds to 8.7 seconds, and each prefill GPU held 3.68 times as many KV blocks. That last figure is the one to read first, because capacity per prefill GPU decides how many long sessions can be in flight before the deployment hits the HBM cliff at all.