11.01 · Concept
Where the tokens actually go
Break a run down by token spend and find the step that dominates it.
Token spend is usually dominated by repeated inputs, agent turns, retrieval context, or tool definitions rather than the final response. A useful trace records tokens, cache behaviour, model choice and latency per step, making it clear whether to cache stable prefixes, trim context, compact history, reduce schemas, or stop loops earlier.
What this lesson answers
- where do LLM agent tokens get spent
- how to find dominant token cost in a trace
- when does prompt caching reduce LLM latency
Notes
A model run spends tokens in more places than the final answer. There are input tokens from the developer instructions, user request, retrieved context, tool schemas, conversation history, and intermediate agent scratch space. There are output tokens from the model’s visible answer, tool calls, structured JSON, retries, and summaries. To find the dominant cost, split the run into steps and record input tokens, output tokens, cache hits, cache misses, model choice, and latency for each step.
The useful mental model is a request trace with a token meter on every edge.
Common questions
- Why is my LLM run expensive when the final answer is short?
- The visible answer may be a small part of the run. Cost can be hidden in repeated system instructions, retrieved documents, tool schemas, conversation history, intermediate tool calls, retries and summaries. Breaking the run into steps usually reveals that large or repeated inputs dominate more than the final text.
- What should I record in a token trace?
- Record each model call as a separate step, with input tokens, output tokens, cache hits, cache misses, chosen model and latency. For agent runs, include tool calls and observations as part of the trace. The goal is to see which edge in the workflow consumes the most tokens or time.
- When does prompt caching actually help?
- Prompt caching helps when many calls share the same stable prefix, such as instructions, examples or tool definitions. It is weakened by changing the early prompt, reordering tools, inserting volatile data too soon, or mixing user-specific content before reusable text. It reduces repeated processing, not every token in the request.