Structured Output
Structured output is generation constrained to a machine-readable shape, usually by masking the model’s next-token probabilities against a schema or grammar during decoding. Unlike prompting the model to produce JSON, constrained decoding makes illegal tokens unsampleable, so the output is syntactically valid for the implemented format.
The problem is that language models emit text, while production systems often need typed data: tool arguments, planner steps, filters, or API payloads. Asking for JSON in the prompt improves the odds, but it does not create a hard boundary. The model can still add commentary, miss a field, choose a wrong enum spelling, or produce a half-valid object that breaks the next component.
Constrained decoding changes the decode loop itself. After the model computes logits for the vocabulary, a parser state representing the schema, regex, JSON grammar, CFG, or finite-state machine determines which tokens are valid next. The sampler then renormalises probability only over that legal set. If a token cannot be consumed by the grammar from the current state, its probability is set to zero.
The trade-off is that the guarantee is syntactic, not semantic. The decoder can force balanced braces, valid field names, and enum spellings, but it cannot prove that an address is real or that a tool call is safe. It can also add latency or reduce batching efficiency if grammar state is complex, differs per request, or is computed on the host in the decode critical path.
Engineers meet structured output in agent serving stacks, function calling, typed SDK responses, database query builders, and workflow routers. A server may compile JSON Schema, Pydantic models, TypeScript-like types, regexes, or function signatures into a token-aware recogniser. The tokenizer detail matters: legal continuation is judged over token byte strings, not just neat source-level characters.
Common questions
- Is structured output just prompting the model to return JSON?
- No. Prompting changes what the model is encouraged to do, but the sampler can still pick invalid tokens. Structured output with constrained decoding changes the set of tokens the sampler is allowed to choose from. Tokens outside the grammar are made unreachable, rather than merely discouraged.
- Does structured output guarantee correct answers?
- It guarantees only that the emitted text matches the implemented grammar or schema. A valid JSON object can still contain the wrong city, an unsafe tool argument, or a poorly chosen enum. You still need semantic validation, authorisation, business rules, and tool-specific checks after decoding.
- Why can structured output affect serving performance?
- The mask over logits is often cheap, but computing legal tokens can become visible when schemas are large, ambiguous, or different for every request in a batch. Host-device synchronisation is a common pitfall. The honest answer is that cost depends on grammar complexity, tokenizer handling, and where the masking runs.
- Why does the tokenizer make this harder than it sounds?
- The grammar is usually written in characters or fields, but decoding happens in tokens. A single token may contain a quote, brace, newline, whole keyword, or partial field name. The recogniser must accept a token only when its byte string can advance the current grammar state without creating an invalid prefix.