Serving

KV working set

Also known as KV cache working set, active KV footprint

In plain English

The KV working set is the attention state needed by the requests and reusable prefixes active over a particular interval.

Technical definition

A KV working set is the collection of attention key and value states used by a workload over a specified interval and memory tier.

Engineering details

Its footprint depends on context lengths, concurrent sequences, shared prefixes, cache precision, and model architecture. Shared blocks should not be counted once per request when measuring physical residency. Allocated cache capacity can also include unused blocks or retained prefixes that are not part of the current working set.

Why it matters

When frequently reused state no longer fits in accelerator memory, the system must transfer, evict, or recompute it. This can produce a sharp latency change as concurrency rises, even though the model weights still fit comfortably.

How to read it in InferenceX

AgentX’s long sessions and overlapping branches stress the KV working set. Compare cache capacity and offload policy alongside throughput; raw accelerator memory capacity alone does not reveal how much useful state a serving configuration can retain.

See the concept in real benchmarks

All articles

DeepSeek V4 Pro on AgentX: B200 vs B300 and the KV Cache Working Set

50% more HBM squeezes out extra throughput, and the per-point telemetry shows exactly where it comes from

AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?

$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200

A Brief Overview of Agentic Workloads

Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure