Input and output sequence length
Also known as prompt length, generation length, 8K/1K
In plain English
Input length is how much the model reads; output length is how much it writes. “8K/1K” means a long prompt followed by a shorter answer.
Technical definition
Input sequence length is the number of prompt tokens supplied to the model; output sequence length is the number of tokens generated in response.
Engineering details
The pair defines the workload shape. An 8K/1K test uses roughly 8,192 input tokens and generates 1,024 output tokens. Long inputs increase prefill work and KV-cache size, while long outputs spend more time in the autoregressive decode loop.
Why it matters
Results from different sequence lengths are not interchangeable. A configuration tuned for short chat prompts can rank differently on long-context summarization or reasoning because compute, memory capacity, and bandwidth pressure shift.
How to read it in InferenceX
InferenceX includes ISL and OSL in chart labels and recipe descriptions. Compare systems on the same workload shape before attributing a difference to hardware or software.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.