Time to first token
In plain English
TTFT is the “thinking…” pause between sending your prompt and seeing the first piece of the answer.
Technical definition
Time to first token is the delay from submitting a request until the first generated token is returned.
Typical unit
milliseconds or seconds
Engineering details
TTFT includes queueing, prompt processing, and any routing or KV-cache transfer before decode begins. Longer prompts generally increase prefill work, while overloaded schedulers can add queueing even when the model computation itself is unchanged.
Why it matters
Users interpret TTFT as how quickly the system begins responding. A system can stream tokens quickly after startup yet still feel slow if requests wait in a queue or prefill competes with decode work.
How to read it in InferenceX
Read TTFT alongside input sequence length, concurrency, and whether prefill is disaggregated. Those details explain why two recipes with similar decode interactivity may begin responses at different speeds.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B