Long context
Also known as long-context inference, long-context serving, long context length
In plain English
Long context means the model is working over a very large amount of conversation, code, or documents at once, which stresses memory far more than raw compute.
Technical definition
Long context describes inference where prompts and accumulated conversation history reach tens or hundreds of thousands of tokens, so KV cache capacity, memory bandwidth, and cache reuse dominate serving behavior.
Engineering details
KV cache size grows with every token in context, and prefill cost grows even faster, so long-context traffic drains HBM well before compute runs out. Coding agents are the canonical source: each turn appends files, tool output, and earlier responses to a session that can span hours. Model architectures respond with sliding window, latent, linear, and sparse attention, while serving stacks respond with prefix caching, KV cache offloading to DRAM and NVMe, and context parallelism.
Why it matters
Hardware and engine rankings measured on short fixed sequences do not transfer to long-context traffic, because the binding constraint shifts from arithmetic throughput to KV capacity and movement. Concurrency limits appear as capacity cliffs, and the systems that handle them define whether agents, retrieval pipelines, and document analysis are economical to serve.
How to read it in InferenceX
InferenceX covers long context from both directions: fixed sequence scenarios pin lengths such as 8K in and 1K out, while AgentX replays multi-turn coding sessions whose contexts grow turn by turn toward realistic agent working sets, sweeping concurrency across the HBM capacity cliff.
Source material
See the concept in real benchmarks
A Brief Overview of Agentic Workloads
Multi-turn sessions, long contexts, and near-total prefix reuse make agentic inference a systems problem, and change what a benchmark has to measure
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
Kimi K3: The Manos, The Mythos, The Legendos
Kimi K3's architecture: compressed memory, attention across depth, latent expert routing, and serving performance