PagedAttention
Also known as paged KV cache, KV cache paging
In plain English
PagedAttention stores the KV cache in small fixed size blocks, like virtual memory pages, so cache memory is not wasted on unused space.
Technical definition
PagedAttention is a KV cache management technique that allocates cache in fixed size blocks addressed through a mapping table, rather than reserving one contiguous region per request.
Engineering details
Contiguous per-request allocation must reserve space for the longest possible output, and most of that reservation is never used. Paging borrows the operating system playbook: cache blocks are allocated on demand as a sequence grows, freed the moment it ends, and shared between sequences with a common prefix through copy on write. Fragmentation drops to near zero, so far more sequences fit in the same HBM.
Why it matters
Introduced by vLLM, this idea unlocked the batch sizes that make continuous batching pay off and became standard across serving engines. Effective KV capacity, not raw memory size, is what bounds concurrency for long context and agentic workloads.
How to read it in InferenceX
All engines in InferenceX recipes manage KV memory in paged or block based form. High concurrency points on long context scenarios such as AgentX are only reachable because paging keeps cache waste small as hundreds of sessions grow and shrink.
Source material
See the concept in real benchmarks
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B