What actually fills the time before the first token?
Surprisingly little of it is generation. The wait stacks connection setup, queueing behind other concurrent requests, and prefill, where the model reads your entire prompt before it may emit anything; prefill cost scales with input length, so a bloated system prompt taxes every single call. Only then does decoding produce the first token. Prompt size is the lever you control most directly: trim what you send, and keep a stable prefix so prompt caching lets repeated context skip reprocessing entirely.
TTFT or tokens per second: which should you optimize?
They answer different complaints. Low TTFT removes the silent pause before anything appears at all, which dominates chat, voice, and copilot interfaces where a user is watching an empty screen. Output speed decides how long a long answer takes to finish once it starts. Users judge responsiveness by the first token they see, not the last, so interactive products optimize TTFT first and treat sustained throughput as the second metric. Track both alongside end-to-end inference latency in llm observability dashboards, measured as percentiles rather than a single flattering average.