10.04 · Concept
Grading the trajectory
Score the steps an agent took: tool choice, order, and recovery.
Agent trajectory grading evaluates the process behind an agent run: tool selection, call order, use of evidence, wasted actions, unsafe behaviour, and recovery from bad results. It separates a lucky final answer from a reliable execution path, using rules, rubrics, or comparisons depending on how constrained the expected workflow is.
What this lesson answers
- how to grade an agent trajectory
- evaluate agent tool choice and order
- score agent recovery after tool errors
Notes
Grading an agent trajectory means judging not only whether the final answer was correct, but whether the path taken was sensible. For an agent, the intermediate steps matter: which tool it chose, what it asked the tool to do, whether it used the result correctly, whether it repeated itself, and whether it recovered when something went wrong. A trajectory grade is therefore closer to reviewing an incident timeline or a pull request than checking a single function return value.
A useful mental model is to treat the agent run as a trace made of decisions.
Common questions
- Why grade an agent trajectory instead of only the final answer?
- A final answer can hide poor behaviour. An agent might call the wrong tool, ignore returned data, expose sensitive context, loop unnecessarily, or reach the right output by chance. Trajectory grading shows whether the agent followed a dependable process, which matters when the same system faces messier inputs in production.
- What should be checked in an agent trace?
- Check whether each tool was appropriate for the current state, whether the tool input was well formed, whether the agent used observations correctly, and whether the sequence made sense. Also look for repeated calls, premature conclusions, unsafe actions, and whether the agent stopped once the task was complete.
- How do you score recovery behaviour in an agent?
- Recovery should be scored by looking at how the agent reacts to failed calls, missing data, or conflicting observations. Good recovery means it recognises the problem, adjusts its plan, tries a sensible alternative, and avoids pretending unsupported facts are true. Partial credit is appropriate when the recovery path is sound but the outcome is imperfect.
Short definition: what is Grading the trajectory?