Curriculum
Module 10
Evaluating agents
Judging a process, not an answer: why output eval falls short, building a test set, trajectory eval, LLM-as-judge, and regression before deploy.
5 lessons · 14 videos · 4h 50m- 10.01
Why grading the answer is not enough
Explain how an agent reaches a right answer the wrong way, and why that matters.
- 10.02
A test set for something non-deterministic
Assemble cases that catch regressions without pretending the output is stable.
- 10.03
LLM as judge, and its failure modes
Use a model as a grader while knowing what it is systematically bad at.
- 10.04
Grading the trajectory
Score the steps an agent took: tool choice, order, and recovery.
- 10.05
The eval that blocks a deploy
Wire evals into CI so a prompt change cannot ship on a hunch.