11.03 · Concept
Using a smaller model where it is enough
Route steps by difficulty instead of sending everything to the largest model.
Tiered model routing sends routine AI workflow steps to cheaper paths first and reserves the largest model for hard, ambiguous or high-risk work. By classifying task difficulty, validating outputs and escalating failures, teams can reduce latency and cost without giving up quality where it matters.
What this lesson answers
- how to route tasks to smaller models
- when should an agent use a larger model
- how to reduce inference cost and latency
Notes
The core idea is that not every step in an AI workflow deserves the largest, slowest, most expensive model. Many requests are routine: classify intent, rewrite a short sentence, extract a date, check whether retrieved context is relevant, or decide which tool to call. Those steps can often be handled by a smaller model, a rules-based function, or a cached answer, while the strongest model is reserved for genuinely hard reasoning, ambiguous instructions, high-risk outputs, or final synthesis.
A useful mental model is a production request router. Before calling a model, ask what kind of work…
Common questions
- What kinds of agent steps can use a smaller model?
- Good candidates are narrow, low-risk and easy to validate: intent classification, short rewrites, date extraction, context relevance checks and tool selection. These tasks usually need consistency more than deep reasoning. If the output shape is predictable and a bad answer can be detected cheaply, a smaller model or deterministic code is often enough.
- How do you decide when to escalate to a larger model?
- Escalate when the task is ambiguous, the input is complex, the confidence signal is weak, retrieved context looks poor, or the result affects something important. A router can start with a cheap path, run validation, then retry with a stronger model when the answer fails checks or falls into a known difficult category.
- Is the largest model always the safest default?
- No. A larger model can improve hard reasoning, but using it everywhere often adds cost and latency without improving the visible result. It can also mask weak task boundaries, poor retrieval or unclear prompts. A safer production design uses the smallest capable path, measurement by route and a reliable fallback for failures.
Short definition: what is Using a smaller model where it is enough?