What actually fills up a context window in production?
Rarely the user’s message alone. A typical call stacks a system prompt, conversation history, tool definitions, and, in a RAG system, a batch of retrieved passages, all competing for the same budget measured in tokens. A window that looks generous on paper fills fast once retrieval, tools, and history are all stacked into the same call, and every extra token costs money whether or not it helps the answer.
Why is a bigger context window not automatically better?
Length is a ceiling on what a model can hold, not a guarantee it reasons well across all of it: models measurably lose track of facts placed in the middle of a very long context, a pattern named the lost-in-the-middle effect and part of the broader degradation researchers call context rot, and bigger windows have not retired it. Testing 18 models across the current generation, Chroma found that “model performance consistently degrades with increasing input length”, even on simple tasks where nothing changed but the length (Chroma, “Context Rot”). This is the practical case for retrieval and deliberate context management over simply pasting in everything available, the discipline covered in context engineering: curate what actually goes in, rather than trusting a large window to sort it out.