Cached input tokens
Also known as cached prompt tokens, cache-read tokens
In plain English
Cached input tokens are parts of a prompt whose earlier processing can be reused, reducing the work needed to read the prompt again.
Technical definition
Cached input tokens are input tokens served using reusable cached model state rather than recomputing their full prefill.
Engineering details
Earlier conversation turns often reappear in later prompts. Whether those tokens hit the cache depends on retained state, routing, and available storage. A provider may price cached input separately from uncached input and generated output, so the three token classes must remain separate in revenue calculations.
Why it matters
A cached token still belongs to the served workload but does not imply the same new computation or sales value as an output token. Potential prefix reuse, measured cache-hit rate, and the price charged for a cache hit describe different quantities.
How to read it in InferenceX
The Rubin article uses total token throughput for several comparisons and distinguishes cached-input, uncached-input, and output prices in its economics discussion. A total-token multiplier cannot be applied directly to an output-only price to estimate revenue.
Source material
See the concept in real benchmarks
Rubin NVL72 Agentic Inference: 67x better Performance per Dollar
Jensen Sandbagging Performance Again, 2x more Annual Profit Per GigaWatt, The More you Buy, The More you Earn, AgentX, InferenceX, Extreme Co-Design
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200