05.04 · Concept
Putting retrieval inside the loop
Let the agent fetch what it needs instead of pre-stuffing the prompt.
Retrieval inside the agent loop means the model asks for context only when its current reasoning needs it. Instead of filling the prompt with guessed documents and outputs up front, the agent can search, inspect results, revise its plan, and fetch again with a narrower purpose.
What this lesson answers
- how should agents retrieve context during reasoning
- when is agentic RAG better than static RAG
- why is prompt stuffing bad for agents
Notes
Pre-stuffing a prompt means deciding in advance which documents, snippets, schemas, examples, or tool outputs the model might need, then packing them into the context window before it starts reasoning. Putting retrieval inside the loop changes that control flow: the model can notice a missing fact, issue a search or database lookup, read the result, update its plan, and repeat if necessary. Retrieval becomes an action the agent can choose, not a one-time setup step the application performs blindly.
A useful mental model is a senior engineer debugging production.
Common questions
- What does it mean to put retrieval inside the loop?
- The agent receives the task, constraints, and available retrieval tools, then decides when it needs more evidence. It can call search, query a database, inspect logs, or read documentation as part of its reasoning process, rather than relying on the application to choose all context before the model starts.
- Why not just put all relevant context in the prompt?
- Stuffing the prompt assumes you know in advance what the model will need. That often adds stale, conflicting, or irrelevant material, which can distract the model and increase cost and latency. Targeted retrieval keeps context tied to the current subproblem and makes the evidence trail easier to inspect.
- When is a retrieval loop worth the extra complexity?
- Use it when the answer depends on intermediate discoveries, changing hypotheses, or multiple sources such as docs, SQL, logs, and search. Static retrieval is usually enough for simple question answering over a known corpus. A loop helps when the agent must decide what to inspect next.