12.03 · Concept
Reproducing a bad run
Replay a failure closely enough to fix it, given temperature and tool state.
A bad agent run is reproducible when the replay preserves the execution context that shaped the failure: prompt, model settings, retrieved data, tool definitions, tool outputs, intermediate messages, retries and external responses. Temperature matters, but the fix usually comes from isolating the boundary where replay diverges from the recorded trace.
What this lesson answers
- how to reproduce a failed AI agent run
- why temperature zero does not guarantee replay
- what to log for debugging agent tool failures
Notes
Reproducing a bad run means capturing enough of the original execution that you can make the system fail again on purpose. With LLM applications, the important inputs are not only the user prompt and code version, but also the model name, sampling settings such as temperature, retrieved context, tool schemas, tool outputs, intermediate messages, retries, timeouts, and any external service responses. A useful reproduction is not perfect determinism; it is close enough that the same bad decision or broken tool interaction appears again.
The mental model is an incident recording for an AI…
Common questions
- What do I need to capture to replay a failed agent run?
- Capture the prompt template, input variables, model name, sampling settings, memory, retrieval results, tool schemas, tool return values, intermediate messages, retries, timeouts and relevant external responses. The aim is not perfect determinism, but enough preserved state to make the same bad decision or tool interaction appear again.
- Does setting temperature to zero make agent failures reproducible?
- No. Lower temperature reduces sampling variation, but it does not freeze provider behaviour, serving differences, retrieval results, tool state, hidden memory, concurrency or external data. Many agent bugs come from those boundaries rather than from sampling alone, so replay needs a recorded execution context, not just a model setting.
- Should replay use live tools or recorded tool outputs?
- Use recorded outputs when the goal is to isolate model or orchestration behaviour from changing dependencies. Use live tools when the suspected bug is in the integration itself. A practical replay often mixes both: freeze stable parts of the failed run, then vary only the boundary you are investigating.
Short definition: what is Reproducing a bad run?