Key takeaway: A large context window is a capability, not an instruction. Filling it because you can is how a working feature becomes an unaffordable one.
The Constraint That Moved
When windows were 4,000 tokens, prompt discipline was enforced by failure — exceed the limit and the request errored. Developers pruned aggressively because they had no choice.
At 200,000 or a million tokens, nothing errors. The prompt grows quietly: the full conversation history, twenty retrieved documents instead of three, an ever-expanding system prompt accumulated through months of edge-case patches.
The request still succeeds. The bill grows linearly with every token, on every call, forever.
Where the Tokens Go
Auditing a mature production prompt usually reveals a similar distribution:
| Component | Typical share | Compressible? |
|---|---|---|
| System prompt and instructions | 10–20% | Yes, substantially |
| Few-shot examples | 15–30% | Often removable entirely |
| Retrieved context | 30–50% | Yes, with better ranking |
| Conversation history | 20–40% | Yes, with summarisation |
| Actual user question | 1–3% | No |
Few-shot examples are the most common source of pure waste. They were added when models followed instructions poorly. Current models frequently perform identically with a clear instruction and zero examples, and nobody re-tested after upgrading.
Retrieved context is the second. Systems commonly pass the top twenty chunks because the retriever returns twenty. Measuring accuracy against k almost always shows a plateau well below that — often at five or six — with everything beyond contributing cost and distraction rather than accuracy.
The Levers Worth Pulling
Prompt caching is the highest-return change available and requires no quality trade-off. Providers charge roughly a tenth for cached prefix tokens. Restructure prompts so the stable portion — instructions, schemas, examples — sits at the front and the variable portion at the end, then enable caching. Applications with large fixed prompts commonly see sixty to eighty percent cost reduction.
Trim k empirically. Plot answer quality against number of retrieved chunks on a fixed evaluation set and pick the knee of the curve rather than the retriever’s default.
Summarise history rather than truncating it. Replacing turns beyond the last few with a compact running summary preserves continuity at a fraction of the tokens.
Route by difficulty. Most requests do not need the largest model. Classify first, escalate selectively, and measure what fraction actually required escalation.
Making It Visible
Cost per request is not a finance metric; it is an engineering metric. Log input tokens, output tokens, cached tokens and model per call, then chart cost per request over time. Prompt bloat is gradual and invisible in aggregate spend, because traffic growth masks it. Per-request cost exposes it immediately.
The Bottom Line
Set a token budget per feature and treat exceeding it as a regression. Enable prompt caching before optimising anything else, then reduce retrieved context to the measured knee rather than the default.
