Is setting temperature to zero enough to reproduce a bad run?

No. Temperature only controls part of model sampling, and even that may not guarantee identical output across provider implementations or serving conditions. Many bad runs are caused by retrieval, tool results, stale external data, retries, hidden memory, or race conditions. Reproduction depends on capturing the whole execution context, not one model parameter.

What should be recorded from a failed agent run?

Record the prompt template, resolved variables, model name, model parameters, message history, retrieved documents, tool schemas, tool arguments, tool outputs, retries, timeouts, memory, and relevant external responses. The guiding rule is to log every boundary where the next state could change. If a dependency cannot be trusted to stay stable, record or mock it.

Should a replay use live tools or recorded tool outputs?

It depends on what you are debugging. Use recorded outputs when you need to isolate the model or prompt path from changing external systems. Use live tools when the suspected bug is in the integration, permissions, latency, or current service behaviour. In practice, engineers often start with recorded outputs, then selectively unmock dependencies.

When is a bad run reproduced well enough?

It is reproduced well enough when the same bad decision, invalid tool call, broken state transition, or incorrect final result appears reliably enough to test a fix. The replay does not need byte-for-byte identical output. It needs to preserve the causal conditions that made the failure happen.

Reproducing a bad run

Reproducing a bad run is the practice of replaying a failed agent or LLM workflow with enough captured context that the same wrong behaviour appears again. It records not just prompts and code, but model settings, retrieved data, tool definitions, tool responses, intermediate messages, retries, timeouts, and external state.

The problem is that agent failures are rarely caused by a single visible input. A user prompt may look harmless, while the actual failure depends on retrieved context, memory, a tool result, a timeout, or a model sampling choice made several steps earlier. Without a reproducible case, debugging becomes guesswork: engineers inspect logs, tweak prompts, and hope the intermittent behaviour returns.

A useful reproduction treats the run as a sequence of state transitions. Capture the prompt template, variables, model name, sampling settings such as temperature, messages, retrieval results, tool schemas, tool outputs, and external service responses. Then rerun from the earliest controllable point. Where live systems would drift, replace them with recorded outputs so the model sees the same context that led to the failure.

The trade-off is that replay is not the same as perfect determinism. Setting temperature to zero is commonly misunderstood as a complete fix, but providers may change serving behaviour, tools may return different data, and concurrency or hidden state can still alter the path. Full capture also adds storage, privacy, and operational overhead, so teams must choose which boundaries to log or mock.

Engineers meet this in trace viewers, eval harnesses, local scripts, notebooks, and regression tests for LLM applications. A failed trace becomes useful when it can identify the divergent step, freeze the relevant tool state, and rerun the scenario after a fix. The goal is to turn “the agent sometimes behaves oddly” into a concrete case that fails before the change and passes after it.

Go deeper

Agents & MCPReproducing a bad runReplay a failure closely enough to fix it, given temperature and tool state.Open the lesson

Common questions

Is setting temperature to zero enough to reproduce a bad run?: No. Temperature only controls part of model sampling, and even that may not guarantee identical output across provider implementations or serving conditions. Many bad runs are caused by retrieval, tool results, stale external data, retries, hidden memory, or race conditions. Reproduction depends on capturing the whole execution context, not one model parameter.
What should be recorded from a failed agent run?: Record the prompt template, resolved variables, model name, model parameters, message history, retrieved documents, tool schemas, tool arguments, tool outputs, retries, timeouts, memory, and relevant external responses. The guiding rule is to log every boundary where the next state could change. If a dependency cannot be trusted to stay stable, record or mock it.
Should a replay use live tools or recorded tool outputs?: It depends on what you are debugging. Use recorded outputs when you need to isolate the model or prompt path from changing external systems. Use live tools when the suspected bug is in the integration, permissions, latency, or current service behaviour. In practice, engineers often start with recorded outputs, then selectively unmock dependencies.
When is a bad run reproduced well enough?: It is reproduced well enough when the same bad decision, invalid tool call, broken state transition, or incorrect final result appears reliably enough to test a fix. The replay does not need byte-for-byte identical output. It needs to preserve the causal conditions that made the failure happen.

Every term