Is constrained decoding slow because every token must be checked?

Usually no. The per-token operation is a mask over the legal next tokens, then normal sampling over what remains. In a well implemented serving stack, that step is small and predictable compared with model execution. If it shows up heavily in profiling, suspect synchronisation, Python callbacks, or inefficient vocabulary handling.

Why can JSON schema output add latency before the first token?

A schema is not directly usable by the sampler. The server may need to lower it into an automaton, handle tokenizer interactions, regexes, escaping, object structure, and legal transitions. That work happens before decoding starts, so a cold schema can inflate admission time and time to first token.

When does grammar caching help structured output serving?

Caching helps when many requests share the same effective constraints. The compiled grammar can be reused, so later requests avoid the admission-time compiler cost. It helps much less when each request embeds unique enum values, generated fields, descriptions that affect the cache key, or per-user regexes.

Serving Agents

07.09 · Walkthrough

What Grammars Cost

Separate the two costs of constrained decoding and find which one is actually hurting: applying a mask per step is cheap and constant, while COMPILING a schema into an automaton is neither, so a fleet with many distinct one-shot schemas pays a tail latency a fleet reusing a few cached ones never sees.

Constrained decoding has two separate costs: cheap repeated masking during generation, and potentially expensive grammar compilation before generation starts. The latency problem usually comes from cache misses on many distinct schemas, not from applying the token mask. Reused schemas behave very differently from one-shot agent-generated constraints.

What this lesson answers

why is constrained decoding slow for agents
what costs matter in JSON schema decoding
how do grammar cache misses affect latency

Notes

Constrained decoding for JSON/schema/tool calls is usually a token-level accept/reject filter: at decode step $t$ , the sampler is restricted from vocabulary $V$ to the legal frontier $A (q_{t}) \subseteq V$ of an automaton state $q_{t}$ , so the logits become $z_{i}^{'} = z_{i}$ if $i \in A (q_{t})$ and $z_{i}^{'} = - \infty$ otherwise, followed by the usual softmax over $A (q_{t})$ . The per-token serving cost is therefore not “understanding JSON”; it is computing or fetching a mask and applying it to $∣ V ∣$ logits, while the non-token cost is compiling a user artifact such as JSON Schema, regex, or EBNF into the automaton…

References

Common questions

Is constrained decoding slow because every token must be checked?: Usually no. The per-token operation is a mask over the legal next tokens, then normal sampling over what remains. In a well implemented serving stack, that step is small and predictable compared with model execution. If it shows up heavily in profiling, suspect synchronisation, Python callbacks, or inefficient vocabulary handling.
Why can JSON schema output add latency before the first token?: A schema is not directly usable by the sampler. The server may need to lower it into an automaton, handle tokenizer interactions, regexes, escaping, object structure, and legal transitions. That work happens before decoding starts, so a cold schema can inflate admission time and time to first token.
When does grammar caching help structured output serving?: Caching helps when many requests share the same effective constraints. The compiled grammar can be reused, so later requests avoid the admission-time compiler cost. It helps much less when each request embeds unique enum values, generated fields, descriptions that affect the cache key, or per-user regexes.

07.08 · Serving AgentsStructured Output 07.10 · Serving AgentsLong-Context Prefill