Validating what comes back
Validating what comes back is the practice of treating model-generated tool arguments as untrusted input, then checking shape, type, permissions, and business rules before any function runs. The model may propose an action, but only trusted application code should decide whether that action is valid and allowed.
Tool calling creates a boundary problem: natural language output is being turned into code with side effects. A model can misunderstand intent, fill gaps with guesses, choose the wrong resource, or produce arguments that look well formed but are dangerous in context. Without validation, a generated request can delete the wrong record, read a forbidden file, query data the user should not see, or trigger an expensive operation.
The concrete pattern is the same as an HTTP API handler. Parse the model-supplied arguments into a known structure, then validate required fields, types, enum values, ranges, identifiers, dates, paths, and resource ownership. Next apply authorisation and domain rules, not just syntax checks. If anything fails, do not call the tool. Return a structured error, ask for clarification, or route to a safe fallback.
The trade-off is more application code and more rejected calls. Strict validation can make an agent feel less fluid, especially when the model nearly got the arguments right. The honest answer is that strictness depends on the tool’s blast radius: reading harmless metadata may tolerate repair, while payments, deletes, file access, and private data require hard gates. Structured output helps formatting, but it is not a trust guarantee.
Engineers meet this in function-calling handlers, agent tool registries, MCP servers, workflow runners, and any place where model output becomes an API request. A schema is a useful first contract, but it should sit beside normal backend controls: validation libraries, permission checks, allowlists, rate limits, audit logs, and clear error paths. The safest mental model is simple: the model is a client, not a privileged subsystem.
Common questions
- Does structured output remove the need to validate tool arguments?
- No. Structured output can make the response easier to parse and more likely to match a schema, but it cannot decide whether the request is authorised, sensible, or safe. A value can have the right type and still refer to another user’s data, an excessive date range, or a forbidden file path.
- What should happen when validation fails?
- Do not execute the tool. Return a clear machine-readable error to the model, ask the user for missing or ambiguous information, or take a safe fallback path. The important part is that failure remains inside trusted application logic, rather than letting the model’s proposed action partially run.
- What is the difference between schema validation and business-rule validation?
- Schema validation checks the basic contract: fields exist, types match, and values fit simple constraints such as enums or ranges. Business-rule validation checks meaning: whether the user may access that account, whether the operation is allowed in this state, and whether the requested resource is on an approved path or allowlist.