01.06 · Concept
Stopping
Implement stopping criteria: EOS token, max tokens, stop sequences. Understand why stopping is a correctness concern, not a convenience.
Stopping criteria decide when an autoregressive generation is semantically complete: an EOS token, a configured token budget, or a matched text delimiter. Correct implementations check after every decode step, keep token and decoded-text state, trim stop delimiters when required, and mark finished requests before they consume more scheduler or cache resources.
What this lesson answers
- how should inference servers stop text generation
- why stop sequences can span multiple tokens
- when should EOS stop autoregressive decoding
Notes
Stopping criteria are predicates evaluated after each decode step to decide whether a sequence is complete: for generated token sequence , stop when . Algorithm: sample next token , append to sequence, update decoded text buffer, evaluate stopping predicates, mark request finished if any predicate is true.
Common questions
- Why is stopping a correctness issue rather than just a UX option?
- If generation continues after an EOS token or a caller-specified delimiter, the server may return text that belongs to another role, a tool-call suffix, or random continuation. The model has already indicated completion, or the API contract has. Emitting beyond that point changes the meaning of the response.
- Why can’t stop sequences be checked only against token ids?
- Stop strings are text patterns, while tokenisation may split a pattern across several tokens. A delimiter such as a role marker can arrive as multiple decoded fragments. The serving code needs a rolling decoded text buffer and must test suffixes after each new token, not just compare the latest token id.
- How should streaming handle stop strings safely?
- Streaming must avoid sending characters that might turn out to be part of a stop delimiter. A robust implementation buffers a short trailing suffix while matching configured stops. Once it knows the suffix is ordinary output, it can flush it. If a stop matches, it usually returns the text before the delimiter.
Short definition: what is Stopping?