13.04 · Concept
Keeping it working after the model changes
Survive a model deprecation or upgrade without re-discovering every prompt.
Model upgrades need production evals, not prompt archaeology. Treat prompts, tool schemas, routing, and model selection as versioned behaviour with a compatibility suite. Run old and new models against representative tasks, compare regressions, and decide whether to adjust prompts, pin, route selectively, or roll forward.
What this lesson answers
- how to test agent prompts before model upgrade
- what breaks when changing an agent model
- how to avoid regressions after model deprecation
Notes
When a model changes, your agent is not just getting a faster or smarter dependency. It is getting a different probabilistic component that may interpret instructions, tools, formats, refusals, and edge cases differently. The way to survive that is to treat prompts, tool schemas, routing rules, and model choice as versioned production behavior, not as one-off text you tweak until it works once.
The core mental model is a compatibility test suite for your agent. Capture representative tasks, tricky past failures, expected tool calls, acceptable answer shapes, latency and cost limits, and…
Common questions
- Why can a better model break an existing agent?
- A stronger model can still interpret your local contracts differently. It may choose tools at different times, format JSON differently, refuse or answer borderline requests differently, or prioritise context in a new way. General capability is not the same as compatibility with your prompts, schemas, and business rules.
- What should an agent eval suite include?
- Include representative user tasks, cases that failed before, expected tool calls, allowed answer formats, safety rules, and limits around cost or latency. The goal is not broad academic measurement. It is to catch the behaviours that matter to your product before a model change reaches users.
- How should a team roll out a model change?
- Run the current and candidate models through the same eval set, then compare failures and improvements. If the new model regresses important cases, update prompts or schemas, keep the old model, or route only suitable traffic to the new one. Make the decision from observed behaviour, not release notes.
Short definition: what is Keeping it working after the model changes?