6. Context windows and the lost-in-the-middle effect

The context window is the maximum number of tokens (prompt + output) the model can process at once. Modern models range from 128K to multi-million tokens.

Two costs to keep in mind: - Prefill: processing the prompt, compute-bound and roughly quadratic in length. - Decode: generating tokens one at a time, memory-bandwidth-bound and served from the KV cache.

Empirically, retrieval accuracy is highest for information at the start and end of a long prompt and dips in the middle — the "lost in the middle" effect. Therefore: - Put instructions and the most critical evidence at the top or bottom. - Retrieve less but better: 5 relevant chunks beat 100 mediocre ones. - Use prompt caching for stable prefixes (system prompt, tool schemas, long documents) to cut latency and cost dramatically.

Gotcha: filling a huge window "just in case" degrades both accuracy and unit economics. Treat context as a scarce, curated resource.