How is trajectory grading different from checking the final answer?

Final-answer grading asks whether the output is acceptable. Trajectory grading asks whether the agent behaved well while producing it. That distinction matters because a correct answer can come from a fragile, expensive, or unsafe path, while an imperfect answer may still show good tool use, evidence handling, and recovery after external failures.

Do I need an exact expected sequence of tool calls?

Not always. Exact sequences work for constrained tasks where there is one intended path. For open-ended tasks, use a rubric that describes acceptable choices and bad behaviours, or compare runs against each other. The more legitimate paths exist, the less useful strict step-by-step matching becomes.

What should a trajectory rubric usually score?

Start with the decisions that create risk or cost: choosing the right tool, forming valid tool inputs, ordering calls sensibly, using observations rather than guessing, avoiding repeated or unnecessary calls, handling errors or missing data, and stopping at the right time. Add task-specific safety checks where misuse would matter.

Grading the trajectory

Grading the trajectory is evaluating the sequence of decisions an agent made during a run, not just its final output. It scores whether the agent chose suitable tools, used them in a sensible order, handled observations correctly, avoided wasted or unsafe actions, and recovered appropriately from errors.

Final-answer checks miss many agent failures. An agent can return the right result after calling irrelevant tools, ignoring contradictory evidence, exposing sensitive context, or looping until it stumbles into a plausible answer. It can also fail the final task while making mostly reasonable decisions after bad data or tool errors. Trajectory grading exists because agent behaviour is a process, and the process often determines reliability, cost, safety, and debuggability.

Concretely, the agent run is treated as a trace: model messages, tool calls, tool inputs, observations, retries, and the stopping decision. A grader inspects each decision against expectations. Was the chosen tool appropriate for the current state? Was the tool input well formed? Did the agent use the returned evidence, change plan after an error, and stop once enough information existed? The grade may be rule-based, rubric-based, or comparative.

The trade-off is effort and subjectivity. Exact path matching is easy to automate but too rigid when several valid routes exist. Rubrics handle flexible behaviour but need careful wording and calibration. Model-based judges can scale review, but they may miss subtle safety issues or reward fluent-looking reasoning. The honest answer is that the right grading method depends on task risk, tool determinism, expected path variety, and how much trace data is available.

Engineers meet trajectory grading in agent evaluation suites, observability tools, regression tests, and post-incident reviews. It is useful when changing prompts, tools, retrieval policies, or orchestration code, because the final answer may stay stable while the path gets worse. Common grading dimensions include tool choice, call ordering, argument quality, evidence use, unnecessary repetition, unsafe disclosure, error recovery, and whether the agent stopped rather than continuing to act.

Go deeper

Agents & MCPGrading the trajectoryScore the steps an agent took: tool choice, order, and recovery.Open the lesson

Common questions

How is trajectory grading different from checking the final answer?: Final-answer grading asks whether the output is acceptable. Trajectory grading asks whether the agent behaved well while producing it. That distinction matters because a correct answer can come from a fragile, expensive, or unsafe path, while an imperfect answer may still show good tool use, evidence handling, and recovery after external failures.
Do I need an exact expected sequence of tool calls?: Not always. Exact sequences work for constrained tasks where there is one intended path. For open-ended tasks, use a rubric that describes acceptable choices and bad behaviours, or compare runs against each other. The more legitimate paths exist, the less useful strict step-by-step matching becomes.
What should a trajectory rubric usually score?: Start with the decisions that create risk or cost: choosing the right tool, forming valid tool inputs, ordering calls sensibly, using observations rather than guessing, avoiding repeated or unnecessary calls, handling errors or missing data, and stopping at the right time. Add task-specific safety checks where misuse would matter.

Every term