Tokenisation at Serving Time
Tokenisation at serving time is the conversion of incoming request text into the exact token IDs a model consumes before inference starts. It determines context use, admission control, scheduling cost, and special-token formatting from the tokenizer output, not from visible characters or frontend length estimates.
The problem is that users send strings, while the model and serving engine operate on token IDs. Character length is a poor proxy: English, long compounds, emoji, accents, and CJK text can expand or compress very differently depending on the tokenizer. A prompt that looks small in a UI can overflow the context window once encoded, especially when the requested generation budget is added.
Concretely, the server normalises or pre-processes the string, applies the model’s tokenizer rules such as BPE, SentencePiece, or Unigram, maps the resulting pieces to integer IDs, and may wrap them with BOS, EOS, or chat template markers. Admission then checks prompt tokens plus requested new tokens against the model limit. The scheduler reserves and batches by these token counts, not by bytes or characters.
The trade-off is that the serving boundary becomes a source of subtle production bugs. Counts are tokenizer-specific, so swapping models or tokenizer revisions changes capacity. Special tokens can be added twice if a client sends formatted chat text and the server also applies a template. Unicode normalisation can change token pieces. Truncation policies are also lossy: preserving system instructions, recent turns, and user content becomes an application choice.
Engineers meet this in OpenAI-compatible gateways, vLLM, TGI, Triton/TensorRT-LLM style preprocessing, custom chat renderers, and request validators. Useful logs include raw character count, token count, tokenizer name and revision, whether special tokens were added, the final rendered prompt, and the token IDs around boundaries. When a request fails, compute whether prompt tokens plus generation budget exceed the model length, then reduce output budget or truncate deliberately.
Common questions
- Why does token count differ from character count?
- Tokenizers learn pieces from training data, not from a fixed character rule. Common English substrings may become one token, while less common words, CJK text, emoji, or differently normalised accents may split into many pieces. The only reliable count is produced by the exact tokenizer and settings used by the serving path.
- Should clients pre-tokenise requests before sending them?
- It depends on the API contract. If the serving system accepts text, the server should be the source of truth because it owns the tokenizer, chat template, and special-token policy. Pre-tokenising on the client can help estimation, but it must use the same tokenizer revision and formatting rules or it will drift.
- What are special-token handling bugs?
- They happen when boundary formatting is applied in the wrong place or more than once. For example, a client may include chat markers and the server may add its own template, duplicating BOS, EOS, user, or assistant markers. The model then sees a different conversation structure from the one the application intended.
- How do I debug multilingual context overflows?
- Log both characters and tokens by language or locale, using the production tokenizer. Compare the ratio rather than assuming a universal characters-per-token estimate. If one language consumes more tokens for the same visible length, adjust quotas, chunking, retrieval limits, and truncation rules by token budget rather than character budget.