Should the returned text include the stop sequence?

Usually no, but it depends on the API contract. Stop strings are often delimiters owned by the protocol, such as a next-speaker marker, so callers expect the content before the delimiter. Some systems expose an option to include it. The important rule is to decide explicitly and apply the same rule before streaming and in the final response.

Why not just stop when the model emits EOS?

EOS only covers the model’s own learned end marker. Applications often need additional boundaries: a chat role marker, a tool-call delimiter, a template separator, or a caller-imposed output limit. The model may not emit EOS at the right time, and user stop strings may be the only reliable way to enforce the surrounding protocol.

Are stop sequences matched against tokens or text?

Stop strings should be matched against decoded text, because the same visible delimiter may be represented by several tokens. A token-only check misses delimiters that cross token boundaries. Token ids are still useful for EOS or configured stop-token ids, but arbitrary strings require a rolling decoded suffix and care around streaming and character boundaries.

Is max-token stopping a correctness mechanism or just a budget limit?

It is both, but it has weaker semantics than EOS or a stop sequence. It prevents unbounded generation and protects scheduler capacity, yet it may cut off a valid answer mid-thought. Callers should record the finish reason, because a length stop often means the result may be incomplete rather than intentionally finished.

Stopping

Stopping is the set of checks an autoregressive inference engine applies after each generated token to decide that a response is complete. It usually combines an EOS token, a maximum output length, and user-supplied stop strings, and it is part of correctness because tokens produced after the boundary are not valid output.

A language model will keep sampling the next token as long as the serving loop asks it to. Without explicit stopping, a completed answer can be followed by role labels, tool-call fragments, training-format delimiters, or unrelated continuation text. This is not just untidy output. If the application promised to stop at a user delimiter or at the model’s end marker, returning later text violates the protocol the caller is relying on.

Concretely, the decode loop appends one sampled token to the request’s token list, updates a decoded text buffer, then evaluates predicates. One check compares the new token with the configured EOS id. Another checks whether the number of generated tokens has reached the configured limit. A text-level check asks whether the decoded buffer currently ends with any stop string. If any predicate is true, the request is marked finished before another model step is scheduled.

The awkward part is that stopping happens at different layers. EOS is token-level, length is sequence-level, and stop strings are text-level. A stop string can be split across several tokens, or even across streamed chunks, so a server often has to hold back a small suffix until it knows whether a delimiter is forming. Matching on raw token bytes is also risky around byte fallback or UTF-8 boundaries; stop matching should use valid decoded text.

The trade-off is between latency, simplicity, and exact protocol behaviour. Streaming every decoded fragment immediately is simple, but can leak part of a delimiter before the server recognises the whole stop sequence. Buffering avoids that but delays a small tail of output. Max-token stopping is a safety cap, not a semantic guarantee. If output ends only because the length limit was hit, the caller should treat that differently from EOS or a stop-sequence match.

Engineers meet stopping in inference APIs as fields such as EOS ids, max new tokens, stop strings, stop token ids, ignore-EOS options, and whether to include the stop string in returned text. In production servers it is tied to scheduler state, not bolted on after HTTP response formatting. Once a sequence is finished, it should free its KV-cache allocation, leave the active batch, and stop receiving sampled tokens.

Go deeper

Inference EngineeringStoppingImplement stopping criteria: EOS token, max tokens, stop sequences. Understand why stopping is a correctness concern, not a convenience.Open the lesson

Common questions

Should the returned text include the stop sequence?: Usually no, but it depends on the API contract. Stop strings are often delimiters owned by the protocol, such as a next-speaker marker, so callers expect the content before the delimiter. Some systems expose an option to include it. The important rule is to decide explicitly and apply the same rule before streaming and in the final response.
Why not just stop when the model emits EOS?: EOS only covers the model’s own learned end marker. Applications often need additional boundaries: a chat role marker, a tool-call delimiter, a template separator, or a caller-imposed output limit. The model may not emit EOS at the right time, and user stop strings may be the only reliable way to enforce the surrounding protocol.
Are stop sequences matched against tokens or text?: Stop strings should be matched against decoded text, because the same visible delimiter may be represented by several tokens. A token-only check misses delimiters that cross token boundaries. Token ids are still useful for EOS or configured stop-token ids, but arbitrary strings require a rolling decoded suffix and care around streaming and character boundaries.
Is max-token stopping a correctness mechanism or just a budget limit?: It is both, but it has weaker semantics than EOS or a stop sequence. It prevents unbounded generation and protects scheduler capacity, yet it may cut off a valid answer mid-thought. Callers should record the finish reason, because a length stop often means the result may be incomplete rather than intentionally finished.

Every term