Context is the budget
Context is the budget means treating an LLM’s context window as a limited working-memory allowance shared by instructions, chat history, retrieved text, tool schemas, tool results, and the model’s reply. Every token included in a run consumes capacity that cannot be used for something else.
The need for context budgeting appears when an agent stops behaving like a single prompt and becomes a long-running system. Files, logs, search results, tool outputs, prior turns, and repeated instructions accumulate silently. When the model fails, engineers often blame reasoning, but the real problem may be that important evidence was crowded out, buried, or diluted by material that was merely convenient to include.
Mechanically, each generation is built from a prompt assembled by the runtime: fixed instructions, the current user request, selected history, retrieved chunks, available tool definitions, tool-call arguments, tool responses, and any scratch or summary state. These are tokenised and placed into the model’s context window. The model then produces output tokens from that input. Accounting means attributing those tokens to components and asking why each component is present.
The trade-off is that context management adds engineering work and can lose information if done badly. Summaries may omit a crucial detail, retrieval may select the wrong chunk, and aggressive trimming may remove constraints that still matter. The honest answer is not always “use less context”; it depends on task risk, evidence requirements, model behaviour, latency, cost, and whether the extra text improves decisions or just adds noise.
Engineers meet this in coding agents, support bots, research assistants, and MCP-style tool integrations. Practical signs include huge tool outputs pasted back into the next turn, unchanged system text sent repeatedly, full files attached when a small region would do, and conversation history growing without purpose. Useful fixes include selective retrieval, stable-prefix caching, structured tool results, state summaries, and per-component token logging.
Common questions
- Is a larger context window the same as memory?
- No. A larger window lets more text be supplied to a single generation, but it is not human memory and does not guarantee attention to every detail. The model can still miss, underweight, or be distracted by information, especially when relevant instructions or evidence are surrounded by stale history and verbose outputs.
- What should be counted in a context budget?
- Count everything sent to or produced by the model: system and developer instructions, user turns, retained history, retrieved passages, tool schemas, tool arguments, tool results, cached prompt sections, summaries, and the expected reply. If a component consumes tokens, it should have an owner, a purpose, and a reason it belongs in this step.
- How do you reduce context without breaking the task?
- Start by removing repeated or low-value material, then replace bulky state with summaries or references where safe. Retrieve only evidence needed for the current decision, shape tool outputs to return relevant fields, and keep stable instructions separate from changing data. For high-risk tasks, prefer preserving source evidence over compressing it too aggressively.