07.08 · Concept
Structured Output
Explain constrained decoding as a mask over the logit vector from module 1: at each step a grammar decides which tokens are legal, and sampling happens only over those. Distinguish it from asking a model nicely for JSON, which is not a guarantee.
Constrained decoding makes structured output a sampling rule, not a prompt convention. After logits are produced, a grammar masks out every token that cannot continue a valid schema, and the sampler chooses only from what remains. That guarantees syntax such as JSON shape, while leaving meaning, safety and business validation to later checks.
What this lesson answers
- how does constrained decoding guarantee valid JSON
- structured output versus asking model for JSON
- where should grammar state live in serving
Notes
Constrained decoding for structured output is a token-level intervention in the decode loop: after the model produces logits , a recognizer for a schema, regex, CFG, JSON grammar, or finite-state machine computes the legal next-token set from the current parser state , and the sampler uses . Tokens outside the grammar get probability exactly zero, not merely lower probability.
References
Common questions
- How is constrained decoding different from prompting for JSON?
- Prompting asks the model to prefer JSON, but the sampler can still choose invalid tokens. Constrained decoding changes the set of tokens the sampler is allowed to emit. Tokens that would break the grammar are assigned no probability, so malformed syntax becomes unreachable rather than merely less likely.
- Does structured output prove the content is correct?
- No. It proves the emitted bytes satisfy the grammar or schema being enforced. It can ensure fields, quoting, braces and enum spellings are valid, but it cannot prove that a city, identifier, amount or tool choice is semantically right. Application validation and authorisation still belong after decoding.
- Why can structured decoding affect serving performance?
- The mask is usually cheap compared with model execution, but the implementation details matter. If grammar checks run on the host, require synchronisation, or differ per request in a large batch, they can enter the decode critical path. Tokenisation also complicates legality, because valid continuations are tokens, not characters.
Short definition: what is Structured Output?