05.01 · Concept
Context is the budget
Account for every token in a long run and find where it went.
Context is the scarce working memory of an LLM run, shared by instructions, chat history, retrieved text, tool schemas, tool results and output. Treat it like a request payload limit: account for what enters, why it is there, what repeats, and what should be trimmed, cached, summarised or retrieved later.
What this lesson answers
- where do tokens go in long agent runs
- how to budget context in LLM workflows
- why larger context windows still fail
Notes
A context window is the fixed working memory an LLM receives for one generation: system instructions, developer instructions, user messages, retrieved documents, tool schemas, tool results, prior conversation, and the model’s own requested output all compete for the same space. Context is therefore a budget, not a filing cabinet. Every extra paragraph, log line, dependency file, JSON schema, or repeated instruction spends tokens that cannot be used elsewhere.
A useful mental model is a production request with a strict payload limit. You would not attach an entire database dump to every API…
Common questions
- What does it mean to treat context as a budget?
- It means every piece of text sent to the model has a cost and must justify its place. Instructions, examples, retrieved documents, tool output and prior conversation all compete for the same limited working area, so useful context should be selected deliberately rather than accumulated by habit.
- Why is a bigger context window not the same as memory?
- A larger window only lets more text be supplied at generation time. It does not guarantee the model will attend to the right details, preserve intent across a long run, or ignore irrelevant material. Too much stale or noisy context can make important information harder to use.
- How do I find wasted tokens in an agent run?
- Break the run into categories: fixed prompts, conversation history, retrieved chunks, tool schemas, tool arguments, tool results and model output. Look for repeated instructions, oversized logs, unused documents, verbose schemas and old turns that could be summarised, cached, trimmed or fetched only when needed.
Short definition: what is Context is the budget?