During generation, a model must attend to the tokens that came before. Without a cache, it would repeat the same attention calculations after every new token. The cache retains those calculations layer by layer, then adds one new set of states for each generated token.
Why is the KV cache fast but expensive?
It avoids repeated work, so decoding can move forward one token at a time. The trade is memory, not free speed. Cache size grows with sequence length, model layers, attention heads, and the number of live requests. A busy server can run out of KV-cache capacity before it runs out of compute.
How is it different from prompt caching?
Prompt caching avoids reprocessing an identical prefix across separate requests, often through a provider-managed mechanism. The KV cache is the immediate working state of a running sequence. They both reward stable context, but they solve different layers of the same cost problem. A KV cache accelerates later decoding, not the initial reading of fresh input. A large context window is only useful if the serving system can hold the required state, and time to first token still includes reading new input before cached generation begins.