10.05 · Concept
The eval that blocks a deploy
Wire evals into CI so a prompt change cannot ship on a hunch.
Blocking evals turn agent behaviour checks into CI gates: prompt, tool, or model configuration changes run against regression cases before merge or deploy. The gate fails when agreed quality rules are broken, making agent changes reviewable engineering work rather than judgement calls based on a local trial.
What this lesson answers
- how to block prompt changes in CI
- what should agent evals check before deploy
- how to fail builds on prompt regressions
Notes
An eval that blocks a deploy is just a test suite for behavior you cannot specify with ordinary unit tests. Instead of asking “does this function return 42,” you ask “does this agent answer correctly, refuse unsafe requests, call the right tools, and avoid regressions on examples we care about.” The important move is making those checks automatic and mandatory: every prompt, tool, or model configuration change runs the eval suite in CI, and the merge or deploy fails if the score drops below an agreed threshold.
A useful mental model is production incident prevention.
Common questions
- What is a blocking eval for an agent?
- It is an automated behaviour check that runs in CI and can fail a merge or deploy. Instead of testing only deterministic code paths, it checks whether an agent gives acceptable answers, refuses unsafe requests, uses tools correctly, and avoids regressions on important examples.
- Do agent evals need to be perfect before blocking deploys?
- No. A blocking eval is a control for known risks, not a proof that the agent is correct. Start with failures you have already seen, common successful paths, and risky edge cases. Make the rules clear enough that engineers understand failures and can fix them.
- What should the CI output include for failed evals?
- The CI job should leave an inspectable record of the failing cases, including the input, the agent output, the expected property or rubric, and the reason it failed. Without that artefact, developers cannot distinguish a real regression from a weak judge or stale test case.
Short definition: what is eval that blocks a deploy?