Interactivity
Also known as generation speed, per-user token rate
In plain English
Interactivity is how quickly one person sees new words appear after the model starts answering.
Technical definition
Interactivity is the rate at which an individual user receives generated tokens during the decode phase.
Typical unit
tokens/second/user (tok/s/user)
Engineering details
It is the reciprocal of time per output token when expressed in compatible units. A response at 50 tokens per second per user emits a new token about every 20 milliseconds after generation begins. Interactivity describes streaming responsiveness, not the delay before the first token.
Why it matters
Different products need different operating points. Voice and interactive coding demand high token rates, while offline summarization can trade interactivity for much more aggregate throughput. Comparing hardware at unmatched interactivity can therefore produce a misleading winner.
How to read it in InferenceX
InferenceX plots tokens per second per user against throughput or cost. Iso-interactivity tables interpolate each system’s Pareto frontier at the same token rate so the comparison holds user experience constant.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18